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ABSTRACT 

An evaluation of voluntary education programs 
sponsored by the military services focused on how servicemembers' 
educational needs are determined and undergraduate education services 
are provided. Using 19 standards originally developed by the Joint 
Committee on Standards for Educational Evaluation, the General 
Accounting Office (GAO) analyzed the services* needs assessment 
processes. At 10 Army, Air Fo;:ce, and Navy installations in Europe 
visitc^ by the GAO, the military services identified the programs or 
courses servicemembers desired to take largely through informal 
day-to-day contacts among students, educational program officers, and 
instructors. This information was supplemented through surve;^s of 
servicemembers. The study found that educational services in Europe 
could be delivered feasibly under the multiple supplier system or 
under the single provider system that the Air Force has adopted. The 
decision as to which of the two systems to use revolved largely 
around questions of competition in the selection of schools and 
advantages or disadvantages of having more than one institution offer 
the same courses. (Appendixes^, amounting to alnost one-half of the 
report include profiles of community colleges snd universities 
providing undergraduate programs in Europe and the evaluation 
standards checklist and evaluation results.) (YLB) 
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Executive Suiranary 



Under Department of Defense (dod) policy, military servicemembers are 
to have educational opportunities similar to those available to other U.S. 
citizens. Consequently, the Army, Navy, and Air Force sponsor pro- 
grams that permit military personnel to pursue educational, vocational, 
and career goals while on active duty. This includes postsecondary edu- 
cation provided to personnel overseas through 2- and 4-year academic 
institutions. During the 1984-85 academic year, more than 150,000 stu- 
dents enrolled in these programs in Europe. 

Concerned about how best to meet the educational needs of military per- 
sonnel, the Congress, through the 1986 Department of Defense Authori- 
zation Act, directed gag to study voluntary education programs 
overseas, gag agreed with the Senate and House Armed Services Com- 
mittees to evaluate how the military services are (1) determining their 
members' educational needs and (2) providing undergraduate education 
services in Europe. 



Background Beginning with the Army in 1949, the military services have made 

^ undergraduate education programs available at their installations in 

Europe. Each installation is required by dgd to annually identify ser- 
vicemembers' educational needs. To meet these needs, the services then 
select academic institutions to provide courses at each installation. 
Today, seven schools offer undergraduate programs at more than 170 
installations in Europe. 

Prior to 1984, the services used a multiple provider system. Each service 
entered into noncompetitive agreements or contracts vvith a number of 
schools to provide all courses in a particular academic program, such as 
liberal arts or aeronautics, at an installation. At any installation with 
more than one school offering programs, each school might offer similar 
core courses (such as English composition). 

In a January 1980 report, the Council on Postsecondary Accreditation 
cited a number of concerns about the quality of the military's program 
in Europe, including unnecessary competition among providers and 
duplication of courses. A dgd review in late 1981 concluded that new 
acquisition procedures were needed for postsecondary education pro- 
grams in Europe. 
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In 1984, the Air Force began to use a single provider system that does 
not permit schools to offer similar core courses as did the multiple pro- 
vider system. Rather, the schools compete for the right to be a sole pro- 
vider of certain courses. Contracts awarded through this new method 
include specific requirements intended to improve the overall quality of 
the postsecondary education program in Europe. 

GAO did its review at 10 Army, Air Force, and Navy installations in 
Europe. With the aid of expert consultants, gao analyzed the services' 
needs assessment processes, using 19 standards originally developed by 
the Joint Committee on Standards for Educational Evaluation (see pp. 
14-15J. 



Although the needs assessment processes at the 10 military installations 
GAO visited varied in scope, focus, and methodology, they generally met 
the standards gao applied. 

With its new delivery system, the Air Force has (1) reduced competition 
for students among schools at the base level, (2) received extended com- 
mitments from the schools to provide education services, and (3) 
included a number of quality-related requirements in its competitively 
bid contracts. During the new system's first year (academic year 1984- 
85), the cost of completing a degree program increased slightly for both 
the service and its members. In part, the cost increase is due to the 
reduced role of 2-year schools (which generally have lower tuitions than 
4-year schools). 



GAO's Analysis 



At the 10 installations gao visited, the military services identified the 
programs or courses servicemembers desired to take largely through 
informal day-to-day contacts among students, education program 
officers, and instructors. This information was supplemented through 
surveys of servicemembers. In general, the services conducted activities 
that GAO consultants considered essential to the needs assessment pro- 
cess. For exar the services identified their audiences and gathered 
valid information with which to assess educational needs (see ch. 2). gao 
did not assess the quality of the education programs provided to satisfy 
the needs identified. 



Results in Brief 



Servicemembers 
Educational Needs Being 
Assessed 
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Effects of New System By adopting the single provider system and awarding contracts to 

schools to provide courses in specific program areas, the Air Force 
aimed to eliminate competition for students at the installation level. 
These contracts also required a 4-year commitment by the schools and 
included quality-related requirements lacking in previous agreements. 

While the Air Force's new system gives the servicemember the opportu- 
nity to obtain the same type of degrees as under the multiple provider 
system, it increased the cost of completing those degrees to both the ser- 
vice and its members. The cost per student enrollment in the Air Force 
rose 8 percent between 1984 and 1985, frorn $135 to $146. During the 
same period, the Army's cost remained the same, $141 per enrollment. 
The Air Force's increase was due in part to the higher tuition charged by 
the 4-year university for lower level (generally freshman and sopho- 
more) liberal arts courses offered the prior year by 2-year institutions. 

The role of 2-year schools in providing services to Air Force members in 
Europe was reduced under the single provider system. In the fall of 
1983, 35 percent of Air Force servicemembers and 38 percent of Army 
servicemembers were enrolled at 2-year institutions. In 1984, with the 
Air Force's new system, its enrollments in 2-year colleges decreased to 
13 percent of undergraduates, while the Army's enrollment in such 
institutions increased slightly to 40 percent. The decrease came about in 
part because the single provider system limits the servicemember's 
choice of school for lower level liberal arts courses. The remaining Air 
Force and Army enroUees were in 4-year institutions (see pp, 26-29). 



ReCOmiHendationS making no recommendations. Whether a service should use the 

single or multiple provider system relates to whether competition is 
needed in the selection of schools and whether there are nonmeasurable 
advantages or disadvantages to having more than one institution offer 
the same or similar lower level courses, gag believes these questions 
should be resolved by policy makers after additional experience is 
gained under the Air Force's single provider system. 



Agency CoininentS concurred with gag's findings and discussed recent actions relating 

to postsecondary education services overseas. It said the Air Force is 
maintaining detailed data which, combined with Army data, will permit 
the services to make an ongoing review of costs and usage by type of 
provider before the next contract negotiations begin (academic year 
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1989-90). DOD stated that the Air Force is negotiating contracts for edu- 
cational programs in the Pacific for academic year 1988-89 and that 
recently issued guidance promotes neither the single nor the multiple 
provider system. Rather, bids are to be solicited from both 2- and 4-year 
institutions. 

GAv, furnished a draft of this report to the two primary undergraduate 
education providers in Europe — the Un?ve>'3ity of Maryland (a 4-year 
institution) and City Colleges of Chicago (a 2-year institution). City Col- 
leges provided comments. While not disagreeing with gag's conclusions, 
It suggested a number of revisions to more clearly reflect the differences 
between the Air Force and Army procurement systems. City Colleges 
concluded that gao had collected enough information to show the supe- 
riority of the Army's multiple provider system, gao believes further 
experience and data collection are needed. 
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Chapter 1 



Introduction 



Over 350,000 active-duty U.S. military personnel are stationed in 
Europe. These personnel along with Department of Defense (dod) civil- 
ian employees and their respective dependents are located at over 170 
major military installations throughout Europe, dod requires each of the 
services to establish educational programs that provide opportunities to 
its personnel to achieve educational, vocational, and career goals, believ- 
ing that military servicemembers should share the same educational 
opportunities available to other citizens. 



To satisfy this requirement,- the respective military services have estab- 
lished (1) the Air Force Education Services Program, (2) the Army Con- 
tinuing Education System, and (3) the Navy Campus. These education 
programs offer studies ranging from basic skills such as improving read- 
ing and communication skills through graduate work. In this report, we 
discuss the portion of the programs dealing with voluntary undergradu- 
ate education provided on military installat »ns in Europe. 



Voluntary Education 
Programs Offered by 
the Military 



The military services have provided voluntary undergraduate education 
services to military personnel overseas since the late 1940s. Over the 
years, these services were acquired from various 2-year or community 
colleges and 4-year institutions through agreements awarded 
noncompetitively. 

As an incentive for servicemembers to participate in the education pro- 
grams, the services provide tuition assistance for up to 75 percent of the 
servicemembers' costs. In 1985, the military provided over $16 million 
in tuition assistance for its members in Europe, dod civilian employees, 
their dependents, and servicemembers' dependents may participate in 
the program on a space-available basis, paying their own tuition. 

A local military installation operates its education program out of an 
education center headed by a dod civilian education service officer (ESQ) 
or education specialist. Additional civilian staff usually consist of educa- 
tion counselors and administrative personnel. Normally, the academic 
institutions have a full-time registrar working at each military education 
center. These personnel assist the ESOs with course schedul^ig, student 
registration, and related tasks. The ESOs have the final authority to 
approve all courses offered and any cancellations that must be made. 
The school registrars act primarily as advisors to the ESQ and represent 
their eniploying institutions. 
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Esos at local military installations have other responsibilities as well. 
Dependmg on the military service, their duties can include academic 
testing, basic skills enhancement training, military training, and educa- 
tional counseling. At some Army education centers, for instance, the 
undergraduate program can amount to less than one-third of eso:>* total 
workload. 

Each school operating in Europe maintains a central staff responsible 
for certifying and assigning faculty, preparing course schedules, raain- 
taining academic records, shipping text books to installations, and gen- 
erally overseeing their progrsjns. The overseas faculty comprises both 
full-time professional educators on leave from stateside campuses or 
permanently living abroad and part-time instructors hired locally. 



In December 1982, the Air Force implemented changes to its voluntary 
education program to correct deficiencos noted in studies completed by 
the Council on Postsecondary Accreditation (copa) and the Office of the 
Secretary of Defense (osd) in 1980 and 1981, respectively. These studies 
basically criticized the military departments for the quality of the vol- 
untary education programs. As a result, in 1983 U.S. Air Force head- 
quarters in Europe issued a request for proposal (kfp) and subsequently 
aware' d contracts that changed the way voluntary postsecondary edu- 
cation services were obtained in Europe. 

The Air Force's new procurement method, referred to in this report as 
the **single provider system,*' was more formal than the procedures pre- 
viously used. According to an Air Forcv. official, prior to the new con- 
tracts there was no competitive bidding by academic institutions to 
provide programs. Instead, the official said, the chief of the Educational 
Program Branch would invite to Europe representatives of certain 
schools he believed could offer programs needed by Air Force personnel. 
The schools that agreed signed a basic agreement or contract and began 
operations where education officials directed. This arrangement is 
referred to in this report as the ^'multiple provider system." 

Under the new Air Force procurement procedures, academic institutions 
had to bid competitively for the right to offer courses in various under- 
graduate program areas such as liberal arts, business, and aeronautical 
technology. To be awarded a contract in one or more areas, an institu- 
tion had to offer a baccalaureate degree. This requirement by the Air 
Force resulted in restricting individual community colleges from compet- 
ing to offer certain general education and liberal aits courses that were 
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part of their individual academic programs. However, a community col- 
lege could bid in consortium with a 4-year school to provide these 
services. 

The University of Maryland now provides all liberal arts and general 
education requirement classroom courses at Air Force installations in 
Europe, and one community college — City Colleges of Chicago — pro- 
vides all vocational and technical courses. Two other 4-year schools — 
Embry Riddle Aeronautical University and Southern Illinois Univer- 
sity — were awarded contracts to offer courses in aeronautics and indus- 
trial technology, respectively. Thus, a student completing a degree in 
vocational education, aeronautics, or industrial technology must take 
the program's required liberal arts and general education requirements 
from the University of Maryland. 

There is, however, an exception to the delivery of lower level (freshmen 
and sophomore) liberal arts courses. City Colleges of Chicago, in addi- 
tion to providing vocational education courses, contracted to provide all 
video cassette courses that would lead to an associate degree awarded 
by that school. These include both lower level liberal arts and vocational 
education courses. 

The Army also attempted to change its procurement of undergraduate 
education services. In November 1984, the Army conmiand in Europe 
issued three rfps that, if implemented, would have resulted in a method 
similar to the Air Force's for providing such services. But the rfps were 
withdrawn because the Congress, responding to community colleges' 
concerns about their diminishing role in providing undergraduate educa- 
tion for the Armed Services in Europe, intervened by amending the 1986 
Department of Defense Authorization Act. Section 1212(a) of the act 
provides for open competition between community colleges and 4-year 
institutions to provide voluntary postsecondary services overseas. As a 
result, schools providing undergraduate services to Army personnel in 
Europe include 

• Big Bend Community College, 

• Central Texas College, 

• City Colleges of Chicago, 

• Embry Riddle Aeronautical University, and 

• University of Maryland. 

See appendix I for profiles of schools providing undergraduate services 
in Europe. 
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The 1986 Department of Defense Authorization Act (Public Law 99-145, 
November 8, 1985) required us to review the voluntary education pro- 
grams of the military services. We were to examine how best to meet the 
educational needs of military personnel, Department of Defense civilian 
employees, and their dependents while ensuring choice of and diversity 
in educational offerings and providers at the most reasonable cost to 
individual participants. 

The act specifically required us to determine 

• the educational needs of members of the Armed Forces of the United 
States and civilian dod employees stationed outside the United States 
and the educational needs of the dependents of such members and 
erriployees and 

• the most effective and feasible means of meeting such needs and the 
cost of providing such services. 

Because of the newness of the Air Force program and the limited first- 
year data available at the time of our review, we agreed with the Senate 
and House Armed Services Committees to examine how the military ser- 
vices are (1) developing their current needs assessments and (2) provid- 
ing postsecondary voluntary education in Europe. 

In conducting our study, we visited 10 military locations in Europe: 5 
Air Force installations— 4 in Germany and 1 in Spain; 4 Army military 
communities in Germany; and 1 Navy installation in Spain. (See table 
1.1.) We judgmentally selected the locations according to (1) the size of 
the installation, (2) whether the installation provided voluntary post- 
secondary education services to remote or geographically separate sites, 
and (3) the portion of the respective service's European program enroll- 
ments or tuition assistance payments the location represented. Army, 
Navy, and Air Force officials said the locations provided a representa- 
tive picture of their voluntary postsecondary education program in 
Europe. 

We evaluated the educational needs assessment process for each of the 
three services. However, we discuss how services were provided in 
Europe for only the Army and Air Force because they acted to change 
their delivery systems. According to Navy officials, the Navy program is 
small in terms of enrollments and the amount of tuition assistance it 
provides compared with the other two services. We did not assess the 
quality of the education programs provided by the systems. 



Objectives, Scope, and 
Methodology 
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Table 1.1: U.S. Military Installations in 
Europe Visited During GAO Review 



Installation 


Country 


Service 


Lindsey Air Station 


Germany 


Air Force 


Ramstein Air Base 


Germany 


Air Force 


Rhein Main Air Base 


Germany 


Air Force 


Torrejon Air Base 


Spain 


Air Force 


Zweibrucken Air Base 


Germany 


Air Force 


Baumholder Community 


Germany 


Army 


Frankfurt Community 


Germany 


Army 


Stuttgart Community 


Germany 


Army 


Wuerzburg Community 


Germany 


Army 


Rota Naval Air Station 


Spain 


Navy 



In addition to interviewing officials of the Departments of Defense, 
Army, Air Force, and Navy, we interviewed officials of academic institu- 
tions that currently provided postsecondary educational services over- 
seas or had expressed an interest in doing so and representatives of 
associations representing these institutions. Our review was done 
between July 1985 and August 1986 in accordance with generally 
accepted government auditing standards. 

We sought comments on a draft of this report from the Department of 
Defense and from the two primary undergraduate education provid- 
ers—the City Colleges of Chicago and the University of Maryland. Writ- 
ten comments were received from dod and the City Colleges of Chicago 
and are included as appendixes II and III. 

Using an evaluation instrument that we developed with the aid of an 
expert in the field of education research and evaluation, we analyzed 
the services' needs assessment processes. The instrument is based on 
evaluation standards developed by the Joint Committee on Standards 
for Educational Evaluation (see ch. 2 and app. IV for a more detailed 
description of the standards). According to the chairman of the commit- 
tee, these standards could be applied to assess the services' needs 
assessment. 

We considered 19 of the committee's 30 standards to be appropriate in 
our needs assessment context and developed specific questions to obtain 
information to determine if the standards were met. Our questions 
focused c:i (1) the types of needs assessment procedures that could be 
followed at military bases, (2) the final product or report detailing 
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results of the assessment, and (3) the use of needs assessment results for 
program planning. 

Our needs assessment instrument was reviewed by nine experts (see 
app, V) in the needs assessment and evaluation fields. The reviewers 
generally agreed that the 19 standards we selected were applicable to 
our purposes and that the questions we developed relative to the stan- 
dards were appropriate for evaluating the services' needs assessment 
processes. Where appropriate, we revised the instrument in response to 
the reviewers' comments. 

The intent of the instrument was to describe each service's needs assess- 
ment process. This included 

• identifying the type and quality of data collected and the extent to 
which information was used in educational planning, 

• determining the degree to which the needs assessment process con- 
formed to our needs assessment standards, and 

• providing the services with prescriptive information on procedures to 
improve the needs assessment process. 

For each militaiy installation we visited in Europe, we completed a 
needs assessment checklist, using information available from documents 
collected and interviews conducted at each installation. (See ch. 2 and 
app. IV for a detailed discussion of our procedures.) 
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Determuiing Undergraduate Educational Needs 
of Military Personnel in Europe 



DOD requires the military services to have their installations conduct a 
needs assessment for use in planning voluntary educational programs. 
The services depend primarily on informal day-to-day contacts between 
students, instructors, and education service officers to identify the edu- 
cational needs of service members. The information obtained is used to 
estimate program and class enrollments for the coming year. The only 
major difference among the Air Force, Navy, and Army planning 
processes is that the Army alone uses a standardized survey instrument 
at all of its European installations to determine these educational 
demands. We found that the services generally included in their assess- 
ment processes the factors considered by experts to be necessary for a 
needs assessment. For example, the services were identilying their audi- 
ences, the information on which assessments were done was valid, and 
the individuals performing the assessments were qualified. 



Educational needs assessment is the foundation for meaningful and cost- 
effective educational programs, accoiding to the dod directive that gov- 
erns voluntary educational programs for military personnel (dod Direc- 
tive 1322.8, Feb. 4, 1980, rev. July 23, 1987). The respective military 
services' regulations implementing their voluntary education programs 
state that needs assessments should be completed for educational plan- 
ning purposes. However, needs assessment varies among the services in 
scope, focus, and methodology. 



Air Force regulations require that educational needs assessments be 
done annually or when a major population change occurs. The role of 
these assessments, however, is limited because many voluntary educa- 
tion needs are identified by Air Force regulations. 

To assess educational needs. Air Force esos survey students, the general 
population of the base, and newly arriving servicemembers as well as 
track students' academic progress. The majority of esos we talked to 
also emphasized the importance of day-to-day informal contact among 
students, instructors, esos, and guidance counselors. Such contacts, 
which occur when students come to the education center to be coun- 
seled, register for courses, and apply for tuition assistance, give center 
staff the opportunity to gather information on students' interests and 
needs. 

The reliability of this data in indicating student preferences was empha- 
sized by Air Force esos. One esq gave the example of an institution's 
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field representative who said she was hearing a lot of interest from her 
students in taking an electronics course. Although the education center's 
survey information showed little interest in the course, the center 
decided to offer the course, and it did attract sufficient enrollments. 

Air Force regulations define many of the needs to be met by base educa- 
tion centers. For example, each center is required to offer the general 
education courses needed to complete an Associate of Applied Science 
degree from the Community College of the Air Force. Also, the regula- 
tion requires that physics, chemistry, and mathematics courses be 
offered at least once a year. 



Army's Needs Assessment The Army's educational needs assessment process is similar to the Air 

Force's. As with the Air Force, a key tool in the process is informal com- 
munication about educational needs or preferences among students, 
instructors, guidance counselors, registrars, and the esq. Information on 
educational needs gathered through daily contacts is the best for plan- 
ning purposes, the Army headquarters officials in Europe responsible 
for educational planning told us. 

In addition, however, the Army introduced in the fall of 1985 a new 
standardized education demand survey, now used by all its installations 
in Europe to gather information on which to base annual course sched- 
ules. The survey examines demand, not need, according to its accompa- 
nying instruction manual Army ESOs had mixed reactions to the survey. 
One of its benefits, according to some esos, was that it gathered new 
information about what time during the day students preferred classes 
to be scheduled. Among negative comments from esos were that it was 
too time-consuming and told them nothing new. 



According to the education specialist at the one Navy location we vis- 
ited, the installation's primary method of needs assessment is the day- 
to-day contact he has with the students. It also does one formal survey a 
year, he said, which produces few surprises. 

At this Navy installation, the academic institutions were also actively 
engaged in needs assessment. One institution's registrar surveys differ- 
ent portions of the base population at least yearly. He also depends on 
direct student input as well as information he develops on where the 
concentrations of upper level students are. Another institution's regis- 
trar said he tracks his school's current students' academic progress to 



Navy's Needs Assessment 
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determine what courses they would need. He also does an annual survey 
of the base's population, he said. 



Evaluation of 
Military's Needs 
Assessment Process in 
Europe 



We reviewed the needs assessment processes at five Air Force, four 
Army, and one Navy location in Europe, using 19 standards. These stan- 
dards were originally developed by the Joint Committee on Standards 
for Educational Evaluation to guide the evaluation of educat.onal pro- 
grams, projects, and materials. According to the committee's chairman, 
the standards are applicable to the evaluation of needs assessments. 
(See table 2.1 for the standards and app. IV for a discussion of them and 
the methodology used in applying them for this study,) 



Table 2>1; Needs Assessment Standards Used in GAP Review 



Category 



Utility 



Definition 



Standards 



To what extent does the needs assessment process focus on the 
appropnate population, address relevant questions, obtain relevant 
information, report It clearly, and assist in the use of the information? 



Audience identification 
Evaluator credibility 
Information scope and selection 
Report clarity 
Report dissemination 
Report timeliness 
Impact 



Feasibility To what extent is the needs assessment process cost-effective and 

workable in real world settings? Is the process operable in the setting in 
which it IS to be applied, and is it realistic, prudent, diplomatic, politically 
viable, and frugal? 

Accuracy To what extent has the needs assessment produced sound information? 

Are the data technically adequate and are the conclusions derived logically 
from the data? 



Practical procedures 
Political viability 
Cost-effectiveness 



Group identification 
Context analysis 

Descnbed purposes and procedures 
Defensible information sources 
Validity 
Reliability 

Systematic data control 
Justified conclusions 
Objective reporting 



Application of Needs 
Assessment Standards 



The Joint Committee on Standards for Educational Evaluation believed 
that the relative importance of individual standards would differ from 
situation to situation and recommended that all standards be considered 
equally important until they could be ranked in the context of a particu- 
lar situation. In addition, determining whether a particular standard has 
been addressed is a subjective decision. 
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To determine if an installation's needs assessment process satisfied the 
requirements for each of th j committee's standards, we developed ques- 
tions designed to measure the presence or absence of these requirements 
at each installation. If we determined that the requirement was present 
at the installation, a "positive" response was recorded for that standard. 
For example, to meet the "information scope and selection" standard in 
the "utility" category, ^ye identified several requirements for the stan- 
dard. They included: 

• Are data collected from groups concerning their preferences for either 
specific programs or courses? 

• Are data on base population characteristics such as educational level/ 
background, military/nonmilitary status, etc., collected? 

If we found all requirements for each standard present on an installa- 
tion, we judged that standard to be met at a 100 percent level. (See app. 
VI for the degree to which each standard was addressed by the three 
services.) 

In the following sections, we provide examples of how ♦^^he services' 
needs assessment processes addressed a selected standard in each of 
two categories: utility and accuracy. 

Utility Category The "information scope and selection" standard under the "utility" cate- 

gory was addressed by the services' needs assessment processes in a 
variety of ways. For example, the Air Force collected data from military 
servicemembers, civilian dod employees, adult family members, stu- 
dents, unit and base commands, guidance counselors, registrars, and 
field representatives. However, the way the information was collected 
varied from base to base. All installations we visited collected data on 
currently enrolled students and used it to determine, among other 
things, course and program preferences and scheduling times. Several 
ESOs tracked student academic progress. 

At Torrejon Air Force Base, for example, student tracking was a 
requirement of the education center. As a result, the schools tracked 
juniors' and seniors' progress to document what courses they were tak- 
ing and needed. The institutions also tracked the course cycle or pattern 
of individual courses to assure that they were offered in the proper 
sequence. The student tracking and course histories enabled the educa- 
tion center's staff to ensure that student program needs were being met. 
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Another method used by Air Force education centers was the collection 
of historical data on individuals. At Zweibrucken Air Base, the ESQ col- 
lected data on the educational level of servicemembers by service 
category. 

At the four Army installations, our review of needs assessments showed 
that they met the ^^information scope and selection" standard by using 
the Army's new voluntary education demand survey, supplementing the 
survey data with information from a number of individuals other than 
currently enrolled students. Each of the four ESOs told us that they inter- 
viewed education center staff, students, base/unit commanders, aca- 
demic institutional representatives, and registrars. To gain a complete 
picture of the education needs on the base, the senior esq at the 
Wuerzburg Community said, the new survey should be supplemented 
with a random sample of the base population. 

The Navy met tlie **information scope and selection" standard at Rota, 
Spain, by using information gathered by the education specialist. For 
example, the education specialist talks to every seaman applying for tui- 
tion assistance and consults with the base's commanding officer before 
adding a program. 

Accuracy Category The "group identification" standard, which we included under the 

"accuracy" category, measured whether the services' needs assessments 
identified the beneficiaries of their educational programs, such as 

• military servicemembers, 

• civilian dod employees, 

• adult family members, 

• students, 

• institutions/field representatives, and 

• unit/base commanders. 

Of the five Air Force installations' needs assessment processes, four sur- 
veyed currently and recently enrolled students and other eligible per- 
sonnel. At Lindsey Air Force Base, for example, the ESQ conducted a 
100-percent survey of all new arrivals on base, reviewed base popula- 
tion data, and periodically assessed students currently enrolled. Most 
ESOS obtained data from currently enrolled students, the schools' field 
representatives, counselors, and/c>r students. At the fifth installation 
this information was collected but never tabulated. 
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At all four Army installations, the needs assessment process included 
surveys of currently and recently enrolled students, and three locations 
also surveyed other eligible personnel. The ESQ at one installation sup- 
plemented the new voluntary education demand survey to more fully 
document such needs. To do so, he reviewed students' past performance, 
assessed the composition of military occupation specialties on the base, 
and administered local interest surveys to students, counselors, and 
servicemembers. 

At the Navy installation, currently enrolled students are also surveyed. 
Also, one academic institution providing service on the base conducts a 
base-wide survey once a year. 
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Unti^ academic year (ay) 1984-85, the Army and Air Force employed 
niultiple provider systems for providing undergraduate services. In 
response to criticisms raised by the Council on Postsecondary Accredita- 
tion (COPA) and the Department of Defense, the Air Force instituted a 
single provider system to reduce the competition for students among 
academic institutions at the base level. The Air Force also received 
extended commitments from schools to provide these education services 
and included a number of quality-related requirements in its competi- 
tively bid contracts. 

During that academic year, the Air Force's change to a single provider 
system resulted in (1) an increased cost of completing a degree program 
for both the service and its members and (2) a reduced role for 2-year 
institutions in providing undergraduate education services in Europe. 



Both the academic community and dod have criticized the military 
departments for the quality of their voluntary educational services and 



The OSD asked copa, which establishes policies and practices in post- 
secondary accreditation, to conduct a study of postsecondary education 
on military bases. In its final report of January 1980, copa listed a 
number of concerns about the military's voluntary postsecondary educa- 
tion program, including 

• unnecessary competition among schools and duplication of programs, 

• institutions without long-term status on many installations, and 

• no system to help assure the quality of educational programs on military 
installations. 

Among copa's recommendations was that osd review and revise its prac- 
tices for procuring the services of educational institutions to provide 
programs at military installations. The Secretary of Defense conducted 
such a review in late 1981 and concluded that all services needed to 
develop and implement new acquisition procedures for the European 
program as soon as possible. 




services. 
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In AY 1984-85, the Air Force responded to the criticisms with several 
changes to its procedures for acquiring education services in Europe. Air 
Force officials said the changes were intended to achieve three goals: 

Decrease competition among academic institutions for the limited 
number of students at the base level, wich the belief this would result in 
lower course cancellation rates; 

Extend the time institutions must commit to providing undergraduate 
education services to prevent them from abruptly terminating their pro- 
grams; and 

Improve the overall quality of the undergraduate programs in Europe. 



Provider Competition at The Air Force adopted a single provider system in part to eliminate com- 
InStaUations Reduced petition for students among schools at the installation level. This compe- 

tition was cited as a problem in both the cx)pa and OSD reviews. In the 
past, program duplication at installations had not been a problem. Air 
Force officials said. The problem arose because in some programs, 
whether offered by a 2- or 4-year school, some similar lower level liberal 
arts courses such as English composition and history were required. At 
an installation with various schools operating, different schools were 
providing lower level courses (generally freshman and sophomore) that 
were basically the same. To avoid this coui*se duplication, the Air Force 
competitively awarded a contract to the University of Maryland to offer 
all undergraduate courses in liberal arts, humanities, and the social/ 
behavioral sciences. These were the courses viewed by the Air Force as 
most likely to be duplicated. 

In total, the Air Force has contracted with four academic institutions to 
provide courses in specific undergraduate and vocational education 
areas (see table 3.1). Under the new system. City Colleges of Chicago, 
Embry Riddle Aeronautical University, and Southern Illinois University 
may offer specific classroom courses in selected program areas. They 
may not, however, offer lower level freshman and sophomore liberal 
arts courses needed to complete a degree in those program areas. Such 
courses may be offered only by the University of Maryland. 



Air Force Responds to 
Criticism Witii New 
Single Provider 
Delivery System 
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Table 3>1: Academic Institutions and Program Courses Offered Under Contract With Air Force 



City Colleges of Chicago 

Vocational technology 
Video cassette program 

including selected lower level 

liberal arts courses 



Embry Riddle 
Aeronautical University 

Aeronautics 



Southern Illinois Univers ity 

Industrial technology 



University of Maryland 

Liberal arts 

Business/ management 
Library orientation 
Computer technology 
Behavioral and social sciences 
Humanities 

Technology and management 



Commitments From Under the new competitive contracts, the Air Force required a commit- 

Schools Extended ^^^^ the part of academic institutions to offer educational services 

for 4 years. Awarding multiyear contracts benefits the servicemember 
and the institution. The commitment deters institutions from abruptly 
terminating programs and, according to an Air Force official, makes 
educational planning and resource allocation easier for the institutions. 

Prior to this change, the negotiated agreements permitted academic 
institutions to discontinue providing educational services at their discre- 
tion. Since 1949, eight institutions have ceased such services in Europe. 
When this happens, students are left with partially completed degrees. 



Improving Quality of To achieve its goal of improving educational quality, the Air Force 

Education included in its new competitively bid contracts a number of specific 

quality-related requirements that had not existed in previous agree- 
ments used to acquire undergraduate education sei-vices. The schools 
now must 

• develop and implement an annual self-evaluation study, 

• assure that their instructors in Europe possess an academic background 
that would qualify them to teach the same courses on their home cam- 
pus, and 

• provide library materials to support courses they offered. 

The lack of such requirements was identified by copa in its 1980 report. 
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Effect of Change to 
Single Provider 
System 



Under the Air Force's single provider system, servicemembers can earn 
the same undergraduate degrees as under the multiple provider system. 
However, the new system (1) increased the cost of completing a degree 
to both service and servicemember and (2) reduced the roie at the Air 
Force's European installations of 2-year schools, which can no longer 
provide certain lower level liberal arts courses. 



Cost to Both Air Force and 
Servicemembers Rises 



The new system has made it mor2 expensive for an Air Force ser- 
vicemember to complete a 2-year degree offered by a community college 
This is because lower level libera! arts courses required to satisfy pro- 
gram requirements must be taken from the 4-y8ar school, which has 
higher tuition rates. For example, if a servicemember could complete all 
63 credit hours needed for an associate degree in civil technology at a 2- 
year school, the total tuition cost would be $3,402. Under the s5ngle pro- 
vider system, however, the total cost increased by to $4,09s9. The 
higher cost to the service assumes the servicemen^ aives 75 per- 
cent in tuition assistance, as shown in table 3.2. ^ ais illustration 
assumes all courses are classroom courses,* 



Table 3*2: Increased Cost of 2-Year 
Degree to Service and Servicemembers 
Under Single Provider System (Academic 
Year 1984-85) 



Type of 

institution 

attended 


Tuition 
(credit hour) 


No. of credit 
hours 
required 
for degree 


Tuition 


Gov't share 
(75 percent) 


CoJt to 
service- 
members 


Combination 
2-year 
4*year 


$54 
71 


22 
41 


$1,188 
2.911 






Total 




63 


$4,099 


$3,074 


$1,025 


2*year only 


54 


63 


3.402 


2.552 


850 



Thus, the increased cost to the government for a student attending a 
combination of 2- and 4-year schools for a degree in civil technology is 
over $500 ($3,074 - $2,552 = $522). 



Tuition Subsidy Per 
Enrollment Increases 



During the first year, tuition per enrollment increased under the Air 
Force's single provider system. Between ay 1983-84 and 1984-85, the Air 
Force average subsidy rose 8 percent from $135 to $146 (see table 3.3). 



* Students may complete their study In libera! arts courses for an associate degree using video cas- 
settes from Uie 2-year Institution and transfer courses from other institutions, which could result in 
lower costs, According to DOD, In fiscal year 108G the Air Force had 4.000 enrollments In video 
cassette courses. 
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Tuition assistance per enrollment paid by the Army during the same 
period remained the same at $141. 



Table 3.3: Average Tuition Subsidy Per 

Enrollment (Academic Years 1Sj3-84 and 
1984-85) 










Service/academic year 


Tuition 
assistance 
(in millions) 


Enrollments 


Tuition 
assistance 
per enrollment 




Air Force: 
AY 1983-84 
AY 1984-85 


$5.4 
6.6 


40.237 
44.805 


$135 
146 




Army: 
AY 1983-84 
AY 1984-85 


7.3 
8.1 


51.735 
57.691 


141 
141 



Two factors contributed to the rise in the Air Force subsidy: (1) the 
schools increased their tuition rates between academic years 1983-84 
and 1984-85 and (2) all lower level liberal arts and general education 
requirement classroom courses were provided by a more expensive 4- 
year school in ay 1984-85. 



Role of 2-Year Colleges Prior to implementation of the single provider system, the distribution 

Reduced enrollments between 2-year and 4-year schools at Air Force installa- 

tions in Europe was somewhat similar to their distribution in the United 
States. For instance, in 1983 and 1984 about 40 percent of undergradu- 
ate enrollments in the United States were at 2-year schools. With the 
advent of the single provider system, the 2-year schools' role in provid- 
ing educational service to Air Force personnel in Europe has greatly 
diminished. In ay 1983-84, 2-year schools accounted for 35 percent of 
the Air Force's European enrollments; in ay 1984-85 this decreased to 13 
percent, as figure 3.1 shows. 

In part, the decrease came about because the single provider system lim- 
its the servicemember's choice of school for lower level liberal art 
courses. Under the Air Force's new system. Air Force personnel cannot 
complete an entire associate degree program at a 2-year school offering 
courses at an Air Force installation. Depending on the program selected, 
the number of courses offered can vary, e.g.: 

• If a servicemember wants to '"'^mplete an associate degree in civil tech- 
nology from City Colleges of Chicago, which is under contract with the 
Air Force, he or she must satisfactorily complete the 63 credits required 
by the school. Under the current contract between the Air Force and the 
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Figure 3.1: Air Force European Enrollments in 2- and 4-Year Schools Compared (Academic Years 1983-84 and 1984 85) 
1963-84 1984-85 




Community 
Colleges 



Four-Year 
Institutions 




Community 
Colleges 



Four-Year 
Institutions 



2-year school, however, the school is authorized to offer only 22 (35 per- 
cent) of the required credits. The remaining credits must be earned 
through courses offered by a 4-year school (the University of Mary- 
land). This example assumes that all courses are classroom courses. 

By contrast, of 60 semester hours required for an associate degree in 
auto technology, 48 (80 percent) can be offered by the City Colleges of 
Chicago. 

An overall decline in the number of vocational courses also contributed 
to the decreased enroUinents in 2-year schools, according to Air Force 
officials. One official said that airmen are just not as interested in com- 
pleting vocational degrees as they once were. We observed from the data 
that the five Air Force installations we visited scheduled few or no voca- 
tional courses. Also, esos told us that the new vocational provider — City 
Colleges of Chicago — was experiencing difficulties starting up programs 
at a number of bases. This was because the school had little experience 
in conducting some of the more traditional vocational programs, e.g., 
welding, electronics, and auto mechanics. Prior to being awarded the 
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vocational contract, the institution had offered mostly lower level gen- 
eral education requirements and vocatioaal programs that did not 
require extensive equipment and laboratories, Air Force officials said. 

We observed these start-up difficulties at one base at which the institu- 
tion scheduled about 10 classes in a term and conducted only one. 
According to the esq at the base, the institution did not put enough 
effort into developing its schedules, resulting in an excessive course can- 
cellation rate. 

The Army, which did not change its delivery system and continues to 
offer servicemembers the option of completing a degree at a 2-year 
scho Ji, experienced an increase of 2 percent in the community colleges' 
portion of total enrollment between academic years 1983-84 and 1984- 
85, from 38 to 40 percent (see figure 3.2). 



Figure 3.2: Army European Enrollments in 2- and 4-Year Schools Compared (Academic Years 1983-84 and 1984-85) 
1983-84 1984-85 




From Army enrollment data for ay 1984-85, we estimate that, had com- 
munity colleges only provided vocational courses (as was the case in the 
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Air Force), their portion of Army enrollments would have decreased 
from 40 to 20 percent. 



Performance by the Air Force and the Army in delivering undergradu- 
ate education in Europe could be measured by such factors as course 
cancellation rates and number of courses conducted. However, the meth- 
ods the services use to determine if a class is to be conducted differed, 
as did the criteria included in these methods. As a result, the comparison 
of the Air Force's and Army's program performance dati was limited 
for academic years 1983-84 and 1984-85. 



Air Force '^^^ Force used an average per class enrollment figure to decide if 

courses were to be conducted. This was determined at each installation 
by dividing the total number of students by the total number of classes 
the students were enrolled in. If the class average was 20 in academic 
year 1983-84 and 15 in 1984-85 the courses were conducted. If the 
installation average was lower, the schools could request the eso to can- 
cel courses to raise the average to the contract minimum. 

Data were not available to calculate what the cancellation rate would 
have been in academic year 1984-85 had the class average remained at 
20, As a result, we were not able to determine a precise percentage 
increase or decrease in cancellation rates between the two years. How- 
ever, we believe that the data are sufficient to support general observa- 
tions about the overall trend, in course cancellation rates. 



Cancellation Rates The Air Force cited a desire to decrease course cancellation rates as a 

reason for using a single provider system. However, the overall course 
cancellation rate increased from 21 to 23 percent between the last aca- 
demic year (1983-84) of the multiple provider system and the first year 
of the single provider system (1984-85). We believe that the Air Force 
probably would have incurred a higher rate in 1984-85 had the average 
er roUment requirement not been reduced from 20 to 15. This is because 
a certain number of classes held in 1984-85, when the average was 15, 
likely would have been cancelled to raise the average to 20. 

Course cancellation rates for vocational education, and upper and lower 
lever undergraduate courses for academic years 1983-84 and 1984-85 
are shown in figure 3.3, While the rate for lower level undergraduate 
courses decreased, that rate as well as the other rates likely would have 



Comparability of 
Program Perf onnaace 
Data Limited 
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been higher had the average class size remained at 20. Also, the higher 
cancellation rate for vocational courses in 1984-85 was due, in part, to 
the start-up difficulties experienced by the new vocational provider (see 
pp. 27-28). 



Figure 3.3: Air Force European Course flB^^^^^^^H 

Cancellations (Academic Years 1983-84 

and 1 984-85) P»rc«nt of CoursM 



40 



30 



20 




For the first four academic terms of ay 1985-86, cancellation rates 
decreased in all categories of courses between academic year 1984-85 
and 1985-86. The vocational education rate was down from 47 to 42 per- 
cent. Cancellations in the undergraduate lower level courses in the first 
year (17 percent) decreased less than 1 percent in the first four terms of 
the second year; the upper level undergraduate cancellation rate (19 
percent) decreased 4 percent in the first four terms of the second year. 
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I Enrollment and Courses Air Force enrollments in undergraduate and vocational education 

? Conducted courses grew by 23 percent between academic years 1983-84 and 1984- 

\ 85, This contrasts with only a 3 percent increase during the preceding 

f academic year (1982-83), Also, the Air Force education centers reported 

an overall 14 percent increase in the number of courses conducted at 
their facilities, including a substantial increase in lower level undergrad- 
i uate courses, as figure 3.4 shows. While the higher student enrollment 

contributed to the increase in lower level undergraduate courses con- 
ducted, we believe that the overall increase would not have been as 
large had the Air Force not reduced its class size criteria. 



Figure 3.4: Air Force Courses Conducted ^^^H^^H 

(Academic Years 1983-84 and 1984-85) 

Numb«r ofCourtM 
2500 




1983-84 




The increase in the lower level undergraduate courses was particularly 
significant, according to Air Force officials, because the majority of stu- 
dents were enrolled in those courses. Also, the officials expressed satis- 
faction that more upper level undergraduate courses were being 
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conducted; this is traditionally the most difficult area in which to offer 
courses because of fewer students enrolling in a particular course. 



Army 



The Army also used an average class enrolhnent minimum to determine 
whether classes would be conducted or cancelled. However, the average 
was developed on a regional rather than individual installation basis 
and differed between schools. Also, schools had the option of conducting 
courses with a lower number of students than the negotiated minimum. 



Cancellation Rate 



The Army's cancellation rates for vocational and lower level undergrad- 
uate courses increased between academic years 1983-84 and 1984-85, as 
showh in figure 3.5. 



Figure 3.5: Army European Course 
Cancellations (Academic Years 1983-84 
and 1984-85^ 
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Enrollment and Courses 
Conducted 



The Army's number of enrollments and courses conducted at its Euro- 
pean installations increased between academic years 1983-84 and 
1984-85, Overall, the Army had a 10 percent increase in enrollments, 
from 72,296 to 79,736, The number of courses conducted on Army 
installations in ay 1984-85 increased over the previous year in all three 
course categories, as figure 3.6 indicates. 



Figure 3.6: Army Courses Conducted 

(Academic Years 1983-84 and 1984-85) 



4000 Numb«r of Coursts 
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1984- 85 



Services' Support Costs 



Army and Air Force cost data in support of postsecondary education 
programs differed because of methods used by the respective services to 
determine support costs. Also, the Air Force could only provide cost 
data on a fiscal year basis. Since the Air Force implemented a single 
provider system, its support costs decreased an estimated 32 percent. 



, EMC 
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The Army's support costs for its undergraduate program decreased 
slightly, by 2 percent, between academic years 1983-84 and 1984-85. 



Air Force support costs included: 



• Transportation costs for moving instructors, their dependents, and 
administrators from the United States to Europe and back and for travel 
within Europe to support voluntary education; 

• Shipping costs for moving instructors' and administrators' household 
goods to and from Europe, and may include transportation of classroom 
and laboratory equipment; and 

• Postage costs for mailing first-class letters, textbooks, and authorized 
office supplies in direct support of classroom instruction. 

In fiscal year 1984, the last yc^r covering the multiple provider system. 
Air Force support costs were about $172,000. In fiscal year 1985, the 
first year the single provider system was in effect, support decreased by 
$35,000 to about $137,000. An Air Force official estimated that logisti- 
cal costs would decrease about $20,000 (15 percent) in fiscal year 1986 
to about $117,000. Air Force combined support costs for both under- 
graduate and graduate schools for fiscal years 1984-86 are shovm in 
table 3.4. 



Table 3.4: Air Force Postsecondary 
Education Support Costs (Fiscal Years 
1984-86) 







Fiscal year 




Category 


1984 


1985 


1986* 


Transportation 


$108,188 


$81,727 


$71,400 


Shipping 


15,000 


14.875 


26,100 


Postage 


49,133 


40.639 


19.500 


Totals 


$172,321 


$137,241 


$117,000 



•Estimated 



For the Army's postsecondary education program, estimated support 
co^ts decreased between academic years 1983-84 and 1984-85. As table 
3.5 shows, overall costs decreased 6 percent while the costs associated 
with the undergraduate program remained relatively stable. 
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Table 3.S: Army Postsecondary Eduction 
Support Costs (Academic Years 1983*84 
and 1984.85) 







Academic year 


Decrease 


Program 


1983-84 1984-85 


Amount Percent 


Undergraduate 


$2,073,010 $2,030,233 


$42,777 2 


Graduate 


1,004,663 866,216 


138,447 14 


Totals 


$3,077,673 $2,896,449 


$181,224 6 



The Army's total logistical and administrative cost was much higher 
than the Air Force's, because the Army's program is much larger and 
because of its method of estimating its program costs. In addition to 
using thi actual cost of travel and shipment of household goods to deter- 
mine the administrative and logistical cost for each school, the Army 
included such factors as (1) housing, estimated at $5,400 per govern- 
ment housing unit occupied, (2) government-furnished equipment from 
the January 1983 General Services Administration supply catalog with 
prices amortorized over 5 years, and (3) government-furnished office 
and storage space at $5.36 per square meter. Also included in the 
Army's costs were estimates for logistical support provided contract 
employees (i.e., commissary and exchange privileges, recreation facili- 
ties, military postal service, officers club, and privately owned vehicle 
license and registration) at $5,463 per contractor employee. 



Education services in Europe can be delivered feasibly under either a 
single or multiple provider system. Comparability of data between the 
services and between academic years for the respective services is lim- 
ited and does not permit a conclusion as to which system is more effec- 
tive. Servicemembers can earn the same type of degrees under either 
system, although they may need to attend courses at two different aca- 
demic institutions under the Air Force system. Also, the Air Force sys- 
tems is somewhat more expensive for the service and sen/icemerabcrs. 

The two systems differ primarily in the role of the academic institutions 
as providers of education services. Under the Air Force system, the 
institutions compete for the right to be sole provider of certain courses. 
Under the Anny system, the same course may be offered by two or more 
institutions which then compete for enrollments among the finite 
number of servicememb'./s at a European installation. Therefore, a deci- 
sion to use a single or multiple provider system revolves largely around 
the questions of whether competition is needed in the selection of 
schools and whether there are nonmeasurable advantages or disadvan- 
tages to having more than one institution offer the same or similar lower 
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level courses. In our opinion, these questions should be resolved by the 
policy makers after additional experience is gained under the Air Force 
single provider system. 



Agency Comments and ^^^^ colleges of Chicago provided written comments on a 

Our Evaluation o t is report. 



DOD Comments ^n its comments (see app. II), dod concurred with our findings and dis- 

cussed recent actions relating to postsecondary education services over- 
seas. DOD stated that the Air Force single provider system needs to go 
through a complete cycle before its efficacy can be evaluated. Also, the 
Air Force is maintaining detailed data that, when combined with the 
Army data, will permit the Air Force to conduct a review of costs and 
usage by provider type before the next contract cycle begins. According 
to ix)D, Air Force data show that the average annual cost increase per 
enrollment for the 5 years prior to the implementation of the new sys- 
tem was 7.8 percem. During the 2 years of the new contract, the 
increase was 7.5 percent, a slight decrease under the new system. 

DOD plans to review the issues dealing with competition in the selection 
of schools and the duplication of courses by 2- and 4-year institutions as 
it monitors the program. Also, tho Air Force is in the process of negotiat- 
ing for new contracts to provide postsecondary education services in the 
Pacific for school year 1988-89. dod stated that the guidance issued to 
the military departments for this acquisition did not promote either the 
single or multiple provider system but solicitation of bids from both 2- 
and 4-year institutions in accordance with procurement procedures. 



City Colleges of Chicago City Colleges did not disagree with the report's conclusions. However, 
Comments City Colleges suggested a number of revisions that it believed would 

clarify the differences between the Air Force and Army's procurement 
systems. City Colleges' comments and our evaluation follow. 

1. City Colleges stated that the report erroneously implies that the Air 
Force single provider system differentiates among providers based upon 
specific program areas. According to City Colleges, the system should be 



O Page 38 ^' i GAO/HRim.1 2 DOD Voluntary Education 

: ERIC 



Chapters 

Meeting Undergraduate Educational Needs of 
Military Servicemembers in Europe 



described as segmenting services by level of provider, because institu- 
tions have to offer a baccalaureate degree to be awarded a contract to 
offer courses in any area other than vocational education. 

As we discuss on pages 11-12, while it was a requirement that a success- 
ful bidder offer a baccalaureate degree, the Air Force's rfp recognized 
that schools in consortium may bid to provide specific courses in one or 
more of the program areas listed in the proposal, such as aeronautics, 
liberal arts, or industrial technology. In fact. City (Colleges, in coiyunc- 
tion with a 4-year institution, submitted a bid in response to the RFP. 

2. City Colleges said that the new requirements included in the Air 
Force's present contracts (such as a long-term commitment to provide 
services and the provision of library materials to support the coui'ses 
offered) could have been incorporated in any procurement method. 
Prior to implementation of the Air Force's single provider system. City 
Colleges' agreement with the Army was amended to incorporate similar 
requirements, the school said. 

We analyzed the two procurement systems but did not independently 
evaluate individual system attributes. However, we agree that the inclu- 
sion of the requirements mentioned are independent of the kind of pro- 
curement system used, and the services' agreements could have been 
amended to include such requirements without changing the method of 
competition or the number of schools providing services in any curricu- 
lum or program. 

3. City Colleges stated that the Air Force and Army contracts define 
minimum class size requirements (i.e., the number of enrolled students 
needed for a class to be conducted) differently. These provisions are 
independent of whether the single or multiple provider system is used, 
City Colleges said, and therefore should not be compared. 

We acknowledge that the Army and the Air Force differ in the way they 
compute contractual class enrollment minimums and revised the report 
accordingly (see p. 29). 

4. City Colleges stated that there is no correlation between the reduction 
in program support costs and the implementation of the single provider 
system as we suggest in the report. They said that the Air Force has for 
some time been implementing a methodical reduction in support services 
to institutional providers, including transportation, shipping, and post- 
age. It also said that any measurable decrease in support costs was 
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because the reimbursement for administrative travel was eliminated 
during the last quarter of the past two fiscal years. 

While the total decrease in program support cost is not attributable to 
the implementation of the single provider system, an undeterminable 
amount is. Under the multiple provider system, the Air Force provided 
support to six differeat institutions; with the implementation of the sin- 
gle provider system the Air Force provides support to only four schools. 
Also, according to an Air Force official in Europe the 2 fiscal years men- 
tioned by City Ck)lleges in its comments are subsequent to the perioa our 
report covers. 

According to Air Force officials, the government has not methodically 
reduced support services, including transportation, shipping, or postal 
services, to institutions. They in fact maintain that although not con- 
tractually required they have provided additional logistical support 
including (1) providing administrative space for the institutions, (2) 
arranging for the authorization of the institutions to use intratheater 
official mail channels; and (3) providing Air Force funded intratheater 
travel for instructor and administrative staff. The Air Force also stated 
that logistical costs have decreased because it only has to reimburse one 
school for travel to various locations to oversee the delivery of courses 
rather than a number of schools offering the same or similar courses. 

5. City Colleges believes there was no significant difference in course 
cancellation rates between the Air Force's last year under the multiple 
provider system and the first year of its sing]** provider system. These 
changes were comparable to the Army's cancellation rates during the 
same period. City Colleges said. 

As we state in the report (see p. 29) and earlier in our evaluation of City 
Colleges comments (see no. 3) we believe that available data were not 
sufficient to determine whether a significant difference existed in can- 
cellation rates between services or between academic years for the indi- 
vidual services. 

6. City Colleges said that our report notes a substantial increase in 
undergraduate courses under the Air Force's new system and a concomi- 
tant decrease in vocational education courses. It suggested that this 
change is the result of the reclassification as "undergraduate" of a large 
number of courses previously classified as "vocational" or otherwise 
unclassified. 
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Reclassification may be one of the reasons for the Air Force's decrease 
in the number of vocational courses and the increase in lower level 
undergraduate courses scheduled. However, as table 3.6 shows, the 
number of vocational courses scheduled in 1984-85 decreased from the 
previous year at a rate less than the increase in the number of lower 
level undergraduate courses scheduled. Also, while the number of Air 
Force vocational classes scheduled decreased 14 percent in 1984-85, 
there was a 36-percent decrease in the courses actually conducted. In 
addition, there was a significant increase in the total number of courses 
offered between academic years 1983-84 and 1984-85. The Air Force 
scheduled 4,028 courses in 1983-84 compared with 4,639 courses in 
1984-85 — a 15-percent incre^e. Therefore, it appears that reclassifica- 
tion was not as significant an issue as was the large percentage of voca- 
tional classes not conducted as scheduled during the period. 



Table 3.6: Air Force Undergraduate ■■■■■■■■■■■■■■■■■■■■■■■■i^ 

Courses Scheduled and Conducted Difference between 

(Academic Years 1983-84 and 1984-85) No. of courses academic years 



AY 1983-84 AY 1984-85 Number Percent 

Courses scheduled: 

Vocational 863 744 (119) (14) 

Lower level undergraduate 2^271 2.928 657 29 

Upper level undergraduate M4 967 73 8 

Totals 4,028 4,639 611 15 

Courses conducted; 

Vocational 608 394 (214) (35) 

L ower level undergraduate 1,846 2,413 567 31 

Upper level undergraduate 737 782 4G 6 

Totals 3,191 3,589 398 12 



7. City Colleges expressed concern that our report said it experienced 
difficulties in starting up vocational programs such as welding and auto 
mechanics. T!\e single provider system created a disruption within its 
structure that could not be instantly corrected, City Colleges said, and 
the problem was worsened by poor forecasting of demand for ' .rses by 
the Air Force. 

While City Colleges has extensive experience in delivering specialized 
vocational courses, as stated above, it agreed that certain start-up prob- 
lems existed. Poor forecasting of demand for vocational courses by the 
Air Force was in part the reason for the high vocational course cancella- 
tion rate. However, as chapter 2 discusses, needs determination is a 
cooperative effort between the services and the institutions. 
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8. City Colleges believes that the information included in our report is 
more than sufficient to show the superiority of the Army's multiple pro- 
vider system. 

We continue to believe that further experience and data collection are 
needed. As we discussed in chapter 1, only 1 year (Ay 1984-85) of pro- 
gram data on the Air Force single provider system was available at the 
time of our review. Further, we could not contrast the Air Force's first 
year data on such performance factors as cancellation rates and courses 
conducted with its previous year data or with Army program data, dod 
in its comments (see app. II) states that the Air Force is maintaining 
detailed data that it believes, when combined with the Army data, will 
permit the Air Force to review costs and usage by provider type before 
the next contract cycle begins. We believe additional Air Force data 
should give the services a basis to evaluate the efficacy of the new Air 
Force system. 
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Big Bend Community College 


Home campus location: 


tviuoco i.ai\c, vvdbiiinyiun 


Began service in Europe: 


1972 


Degrees offered: 


Associate in Applied Science 


Programs offered. 


Auto mechanics 




Drafting 




Electronics 




Food service 




Construction technology 




Recreation 




Welding 




Photography 




Engineer technology 


Student enrollment for AY 1985-86: 


1,137 


Faculty for AY 1985*86. 




Full-time 


0 


Part-time 


75 


Tuition rate per semester hour for 


$34 without lab 


AY 1985-86: 


50 with lab 


Accreditation. 


Northwest Accreditation Association 


(continued) 
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Central Texas College 



Home campus location: 
Began service in Europe: 
Degrees offered: 
Programs offered: 



Student enrollment for AY 1985-86. 

Faculty for AY 1985-86: 
Full-time 
Part-time 

Tuition rate per semester hour for 
AY 1985-86: 

Air Force 

Army 

Navy 



Accreditation: 



Killeen, Texas 
1972 

Associate degrees 

Applied management 
Automotive service and repair 
Automotive body repair 
Child development 
Communications electronics 
Computer electronics 
Computer science 
Drafting and design 
Food service management 
Law enforcement 
Office management 
Photography 
Welding 

Administrative/secretanal 
Air conditioning and refrigeration 
Consumer electronics 
Diesel mechanics 
Hotel/motel management 
Information systems management 
Small gas engines 
Telecommunications 

Applied management with technical options 
32.848 



130 
284 



58 

44 without lab 

66 with lab 

40 welding classes 

Southern Association of Colleges and 
Schools 



(continued) 
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City Colleges of Chicago 

Home campus location: 

Began service m Europe; 
Degrees offered: 

Programs offered 



Student enrollnient for AY 198S86- 

Faculty for AY 1985-86. 
Full-time 
Part-time 

Tuition rate per semester hour for 
AY 1985-86- 
Army 



Air Force 



Navy 



Accreditation: 



Chicago. Illinois 
1969 

Associate m Arts 
Associate in Applied Science 

Accounting 
Architectural drafting 
Architectural technology 
Art 

Automotive technology 
Business administration 
Civil technology 
Data processing 
Electronics 

Emergency medical services 
Engineering technology 
Fire science 
Food management 
Health facilities management 
Law enforcement 

Approximately 33,900 



46 
210 



$23 Chicago resident 

42 VicJeo cassette 

36 Audio cassette 

65 Emergency medical services 

60 Data processing 

50 All other courses 
$23 Chicago resident 

54 Vocationc! without lab 

65 Vocational with lab 

36 Video cassette (no instructor) 
$23 Chicago resident 

74 Emergency medical services 

65 Data processing 

65 Electronics 

39 Video cassette (without instructor) 
45 Video cassette (with instructor) 
54 All other courses 

Northern Central Association of Colleges and 
Schools 

(continued) 
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Embry Riddle Aeronautical University 



Home campus loceilinn* 


Bunnell, Florida 


Began service in Europe: 


1970 


Degrees offered: 


Associate in Science 


Bachelor of Science 




Masters degrees 


Programs offered: 


Aircraft maintenance 


Professional aeronautics 




Aeronautical studies 




Aviation maintenance management 




Aviation business administration 




Aviation technology 


Student enrollment for AY 1985-86: 


5.261 


Faculty for AY 1985-86: 




Full-time 


4 


Part-time 


120 


Tuition rate per semester hour for 




AY 1985-86. 




Air Force 


$95 


Army 


80 


Navy 


85 


Accreditation* 


Southern Association of Collegei? and 




Schools 


Southern Illinois University 


Home campus location: 


CarbondaU*. Illinois 


Began service in Europe- 


1982 


Degrees offered 


Bachelor of Science 


Programs offered 


Industnal technology 


Student enrollment for AY 1985-86 


750 


Faculty for AY 1985-86: 


8 


Full-time 


Part-time 


16 


Tuition rate per semester hour for 


$109 


An 985-86 





Accreditation. North Central Association of Colleges and 

Schools 

(continued) 
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University of Ut Verne 



Home campus location: 
Began service in Europe: 
Degrees offered: 



Programs offered: 



Student enrollment for AY 1985-86: 

Faculty for AY 1986-86: 
Full*time 
Part-time 

Tuition rate per semester hour for 
AY 1985-86: 

Accreditation 



Sepulveda, California 
1975 

Associate in Arts 
Associate of Science 
Bachelor of Arts 
Bachelor of Science 
Masters degree 

General studies 
Italian studies 
Criminology 

Communications technolc^v 

Digital technology 

Information systems technology 

Accounting 

Behavioral science 

Business nanagsment 

Business administration and economics 

Computer science and computer engineering 

Health care management 

Marketing 

Mathematics 

Political science 

Psychology 

Social science 

Sociology 

2,186 



9 
37 

$65 



Western Association of Schools and Col leges 

(continued) 
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University of Maryland 



Home campus location: 
Began service in Europe: 
Degrees offered: 

Programs offered: 



Student enrollment for AY 1985-86' 

Faculty for AY 1985-86- 
Full-time 
Part-time 

Tuition rate per semester hour for 
AY 1985-86: 

Air Force 

Army 

Navy 

Accreditation: 



College Park, Maryland 
1949 

Associate in Arts 
Bachelor of Arts 
Bachelor of Science 
Masters degree 

General curriculum 

Business and management 

Paralegal studies 

Anthropology 

Art history 

Computer studies 

Economics 

English 

Foreign languages 
Government and politics 
History 

Law enforcement/criminology 

Management studies 

Philosophy 

Psychology 

Sociology 

114,799 



747 
913 



$71 
73 
71 

Commission on Higher Education of the 
Middle States Association of Colleges and 
Schools 
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ASSISTANT SECRETARY OF DEFENSE 



WASHINGTON. O C 20301.4000 



•» ORCE MANAGEMENT 
AND PERSONNEL 



2 2 SEP 1987 



Mr. Prank C. Conahan 
Assistant Conptroller General 
National Security and 

International Affairs Division 
U.S. General Accounting Office 
Washington, DC 20548 

Dear Mr. Conahan: 

This is the Department of Defense (DoD) response to the 
General Accounting Office (GAO) draft report, "DOD VOLUNTARY 
EDUCATION: Determining and Meeting Postsecondary Educationii.l 
Needs in Europe," dated July 10, 1987 (GAO Code 104566/OSD Case 
7345). The Department concurs with the report findings. The 
Department has taken some recent actions in this area, as 
follows : 

The Air Force, as the executive agent for the Services 
educational programs in the Pacific, is in the process of 
negotiating that con.ract for the school year 1988-39. 

On April 3, 1987, the Assistant Secretary of Defense (Force 
Management and Personnel) issued guidance to the Military 
Departments for acquisition of voluntary education programs for 
military personnel overseas, r.ie guidance does not promote 
either the single or multiple provider system; rather, it 
solicits from both two- and focr-year institutions, in accordance 
with established acquisition procedures. 

The GAO findings on changes in costs cancellation rates and 
enrollments between the Air Force and Army programs indicate that 
both systems took into account the educational needs of the 
Service members and the Department of Defense, and that there was 
no appreciable difference between the two programs. The GAC also 
concluded that competition in the selection of schools and 
duplication of couTses at installations are DoD policy concerns. 
The DoD plans to review thewe Issues prior to the 1989-90 
contract negotiations, in addition, the Department will monitor 
the areas examined by the GAO on an on-going basis. 

The detailed DoD comments on the GAO findings are provided 
In the enclosure. The Department appreciates the opportunity to 
comment on this draft report. 




Principal Deputy 



Enclosure: 
As stated 
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GAO DRAFT REPORT - DATED JULY 10, 1987 
(GAO CODE 104566) OSD CASE 7345 

"DOD VOLU^rrARY EDUCATION: DETERMINING AND MEETING POSTSECONDARY 
EDUCATION NEEDS IN EUROPE" 

DEPARTMENT OF DEFENSE COMMENTS 



* * * * * 



FINDINGS 



Now on pp.2, 1C-1 1,22-23. 



FINDING A ; Voluntary Education Programs Offered By The Military , 
The GAO reported that, under Department of Defense (DoD) policy. 
Military Service members are to have educational opportunities 
similar to those provided other U.S. citizens. The GAO noted 
that, begi ling in the late 1940s, voluntary undergraduate 
education services have been provided to military personnel 
overseas. The GAO found that, as an incentive for Service 
members to participate, the Services provide tuition assistance 
for a minimum of 75 percent of the Service members' cost, and in 
1985 provided over $16 million for tuition assistance in Europe. 
The GAO reported that prior to 1984, the Services used a multiple 
provider system. The GAO found, however, that the Office of the 
Secretary of Defense (OSD) requested the Council on Postsecondary 
Accreditation (COPA) to conduct a study of the DoD program, and 
in its January 1980 report, COPA listed a number of concerns, 
including: 

- unnecessary competition among providers and duplication of 
programs ; 

- institutions on many installations with no long term status; 
and 

- no system to help assure the quality of educational programs. 

The GAO noced that among the study recommendations was that the 
OSD review and revise itd practices for procuring the services of 
institutions providing this postsecondary education. The GAO 
reported that in late 1981, the Secretary of Defense conducted 
the recommended review and concluded thet new acquisition 
procedures should be developed and implemented for the European 
educational program as soon as possible. (pp. 1-2, pp. 8-9, pp. 
25-26/GAO Draft Report) 

DOD RESPONSE : Concur. It shoulr'i be noted, ".^wever, that 
Title 10, United States Code, paction 2007, states that the 
Service? may not pay irwre than 75 percent of the costs of an 
educational institution for a member enrolled during off-duty 
time. Tide 10 provides an exception for enlisted members, E-5 
or higher with less than 14 years service. The/ may receive up 
to 90 percent in tuition assistance. The draft report does not 
accurately reflect the statute, which sets maximum amounts rather 
than minimum levels as stated in the draft report. The DoD 
responses to Findings B and C are also germane. 
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FINDING B i Air Force Responds To Criticism With New Single 
Provider Delivery System , The GAO found that, in December 1982, 
the Air Force implecented changes to correct deficiencies noted 
by the above studies (which basically criticized the Military 
Depart'aents for the quality of the voluntary education programs). 
The GAO reported that in order to (1) decrease course 
cancellations, (2) extend institutions* commitments, and (3) 
improve overall quality, in 1980, the Air Force started to use a 
single provider system that does not permit schools to offer 
similar courses: instead, the schools compete for the right to 
be the sole provider of certain courses. The GAO reported that 
the Air Force contracted with four academic institutions to 
provide courses in specific ac^Cemic areas, and that the 
contracts included a number of speciiic requirements from the 
schools , including: 



- a 5-year commitment; 



" an annual self-evaluation study; 



Now on pp. 2-4. 1m;^ 23-24. 



- assurance that instructors in Europe oos^ens the academic 
backgrounds that */ould qualify them to tcv^h the same course on 
the home campus; and 



- tha proviisior 



library nateria?.:: in support of courses. 



Tlie GAO observed that awarding wulti-year contracts benefits both 
thft Service mor^oer and Cne ina' i tut ion by ensuring that 
institutio'^B cannot diocontinue educational services, thereby 
leavin»^ .S^iw v'ice m^ abors with partially completed degrees, and 
(according to &n k.^r Fcxc^j officer) by making educational 
planning easiar for the institution, (pp. 2-5, pp. 9-11, pp. 
26-29/GAO Di:ift Roport) 

rOD RESPOKSS : Concur. T,ie 1980 version of DoD Directive 1322 c, 
"Voluntary Education Prograois for Military Personnel," set 
standards for educational institutions conducting programs on 
installations. The Air Force followed these guidelines in 
selecting the institutions for its single provider system. In 
order to ensure i.hat DoD continues to provide criteria for 
quality control, DoD Directive 1322.8 was revised and reissoed cn 
July 23, 1987. Enclrsures 3 and 4 of the Directive provide 
standards for obtaining and selecting educational programs at 
installations . 



FINDING C : Cr igressiongl Action And GAO Review . The GAO 
reported that the Army aloo attempted to change its procurement 
method and issued three reque(»ti] for proposals (RFPs) that would 
nave resulted in a me!:hod similar to the Air Force. The GAO 
found, however, that the RFPs were withdrawn because the 
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Now on pp. 13-15. 



Congress, responding to community college concerns about their 
diminishing role in providing education for the Armed Services in 
Europe, amended the 1986 Defense Authorization Act to provide 
open competition between community colleges and 4-year 
institutions. The GAG reported that the Act also required the 
GAG to review the voluntary education programs of the Military 
Services. The GAG further reported that, because of the newness 
of the Air Force program and the limited first year data 
available at the time of its review, the Arued Services 
Committees agreed the GAG >ihould evaluate: 

- how the Military Services are developing their current needs 
assessments ; and 

- how postsecondary voluntary education is being provided in 
Europe. 

In carrying out its review, the GAG visited ten installations in 
Europe, five Air Force, four Army and one Navy. The GAG noted 
that it had attempted to develop a data collection instrument for 
evaluating program performance, but found that much of the data 
considered Critical (for instance, the characteristics of 
faculty) was not available and so it did not evaluate performance, 
(pp. 11-14/GAG Draft Report) 

DOD RESPONSE i Partially concur. In selecting institutions to 
conduct programs on the installations, the Services rely on 
civilian accrediting associations to ensure that institutions 
meet high quality standards for academic programs and support 
services. In addition, DoD Directive 1322.8, "Voluntary 
Education Programs for Military Personnel," enclosure 4, sections 
D, I, and J requiras the institution home campus to participate 
in faculty selection for the installation program, and maintain 
regular contact between institutions overseas and home caopus 
personnel. The Department does not require that information on 
faculty workloads and credentials be maintained by the DoD 
Education Centers, either in the United States or overseas. 
Faculty qualifications are regulated by institutions and 
accrediting associations. Data relating to the standards and 
qualifications for faculty teaching in Europe are required by the 
Service contracts to be available from the institution 
representatives. 



FINDING D: 



Needs Assessment Provides The Basis For Educational 



Planning . The GAG reported that a local 
operates its education program out of an 
by a DoD civilian education service offic 
specialist. The GAG observed that DoD Di 
that educational needs assessment is the 
meaningful and cost-effective educational 
Military Services, in their implementing 
needs assessm^mts should be carried out. 



military installation 
education center headed 
er (ESG) or education 
rective 1322.8 states 
foundation for 

programs, and that the 
regulations, state that 

The GAG found that, at 
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Now on pp. 3, 10. 1608. 



Now on pp. 3. 1405, 16-21. 



the ten installations it visited, the Military Services 
identified the programs or courses Service members desired to 
take largely through informal aay-to-day contacts among students, 
EPOs, and instructors. The GAO further found that the only major 
difference among the Air Force, Navy and Army planning processes 
is that the Army uses a standardized instrument to determine 
educational needs at all of its European installations. (The 
GAO noted that the Army introduced a new st^tndard survey in the 
fall of 1985,) The GAO also found that Air Force regulations 
require that an assessment be done annually or when a major 
population change occurs, but that the role of these assessments 
is limited because many voluntary education needs (such as 
general education courses needed to complete an Associate of 
Applied Science degree from the Community College of the Air 
Force) are specified by regulation. (p. 4, p. 9, pp. 16-18/GAO 
Draft Report) 

POD RESPONSE ; Concur. 

F INDING E t Needs Assessment Standards Used In GAO Review . The 
GAO used 19 standards — originally developed by the Joint 
Committee on Standards for Educational Evaluation — to guide the 
evaluation of educational programs, projects, and materials. The 
GAO reported that thes^e standards were directed at evaluating to 
what extent the needs assessment process focuses on the 
appropriate population, addresses relevant questions, obtains 
relevant information, reports it clearly, and assists in tl)e use 
of the information. The GAO also indicated that these standards 
addressed the extent to which the needs assessment process was 
cost-effective and workable in real wor' 1 settings and operable 
in the setting in which it was to be applied: Was it realistic, 
prudent, diplomatic, politically viable, and frugal? The 
standards the GAO ur.ed were also designed to show to what extent 
the needs assessment produced sound information, techuicMly 
adequate data and conclusions derived logically from the data. 
At each installation the GAO interviewed program officials 
focusing on the types of needs assessment procedures followed at 
military bases, the fin^l product or report detailing results of 
the assessment, and the use of needs assessment results for 
program planning. The GAO also collected relevant program data, 
(p. 3, pp. 14-15, pp. 19-21/GAO Draft Report) 



DOD RESPONSE; 



Concur. 



FINDING F: Services' Needs Assessments Generally Addressed 
Essential Aspects . Using its standards as a checklist at the ten 
instailauiono, they found that the Services met these standards 
to a high degree. As examples, the GAO cited the "information 
scope and t action** standard in the "utility" category and the 
"group It' .cation** standard in the "accuracy" category. The 

GAO '» for the first, the Air Force needs assessment 

pxc : 3d data from Military Service members, DoD 

civ^ix-.. /iss, adult family memb^jrs, students, unit and 

>'^se comA . '\nce counselors, registrars, and fi<\d 
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Now on pp. 3. 18-21.74. 



Now on pp. 3-4. 22. 25-29. 



representatives. In addition, the GAG found that the Army 
installations also met this standard by using their p iw voluntary 
education demand survey and supplementing the r*urvey daf.a with 
information from interviews with education center staff, 
students, base/unit commanders, representatives of academic 
institutions, and registrars. The GAG found the standard was met 
by the Navy at Rota, Spain, by using Information gathered by the 
education specialist. For example, the education specialist 
talked to every seaman applying for tuition assistance and 
consults with the base commanding officer before adding a program. 
With regard to the second standard, the GAG found that at most of 
the installations this standard was met by surveying currently 
and recently enrolled students and other eligible personnel. The 
GAG concluded from its survey that the factors considered by 
experts to be necessary for a needs assessment were generally 
included in the Military Services* assessment processes. (pp. 
3-4, p. 161, pp. 21-24, pp. 65-66/GAG Draft Report) 

POD RESPONSE : Concur. 

FINDING G ; Effect Of Air Force Change To A Single Provider 
Systc/n . The GAG reported that, while the new Air Force system 
provides the Service members the opportunity to obtain the same 
type of degrees as under the multiple provider system, it has 
increased the c«-»st of completing that degree to both the Service 
and its members. The GAG found that the cost to a Service member 
to complete a 2-year degree offered by a community college has 
increased. Also, the GAG found that the cost per student 
enrollment to the Air force rose 9 percent in 1985 — to $146 from 
$135 in 1984. During the same period, the Army cost remained the 
same at $141 per enrollment. The GAG concluded that the Air 
Force increase was due in part to the higher tuition rates 
charged by the 4-year university for lOwer level liberal arts 
courses that were offered the prior year by 2-year institutions. 
In addition, the GAG found that, under the single provider 
system, t>e role o^ 2-year schools in providing services to Air 
Force members in Europe was reduced. The GAG cited, as an 
example, in the fall of 1983, 35 percent of Air Force Service 
members and 38 percent of Army Service members were enrolled at 
2-year institutions; while in 1984, the Air Force enrollments in 
2-year colleges decreased to 13 percent of undergraduates and th^ 
Army enrollments in such institutions increased slightly to 40 
percent. (p. 3, pp. 5-6, p. 25, pp. 29-34/GAG Draft Report) 

POD RESPONSE ; Partially concur. The DoD supports the Air 
Force effort to develop and maintain a high quality educational 
program that provides access to educational programs for Servile 
members overseas. Air Force data show that, for the five-year 
period prior to the new system, the average annual cost increase 
per enrollment was 7.B percent; during the two years of the new 
contract, the increase was 7.5 percent, a slight decrease under 
the new system. The GAG used the least popular program^CC ivi 1 
Technology) to describe the number of courses required by the 
four-year school as opposed to the two-year school. A better 
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example would be the associate degree in Auto Technology. In 
this program, 60 semester hours are required, and 48 of those are 
offered by the two-year school. Rather t'lan use a specific 
degree program as an example, a more appropriate analysis could 
be developed using a sample of several degr<i<s programs. 

The video tape program offered by the City Colleges of Chicago 
o£fers all the general education courses for the baccalaureate 
degree and also offers a complete associate degree. Recent data 
oiiow 4,000 enrollments in FY 1986, for a cost of ?33 per semester 
hour, as opposed to the higher rates for classroom based courses 
conducted by both two- and four-year institutions. Over 8,000 
enrollments are projected for this program in fV 1987. 

In its calculations, the GAO has not taken into account the fact 
that the average enlisted student has 15 semester hours with the 
Community College of the Air Fores, thereby reducing the cost of 
the associate degree for both the :iiraan and the Air Force. 

The Air Force provider system needs to go through a complete 
cycle before its efficacy is decided. The Air Force is 
maintaining detailed data in line with the GAO findings. That 
data base combined with the Army data will permit the Services to 
conduct an on-going review of costs and usage, by type of 
provider, before the nej:t contract cycle begins. 

FINDING H ; First Year Data Si ws Many Similarities Between Army 
And Air Force Performan ce^ The GAO observed that, although 
limited Dy a one-year result. Air Force implementation of 
the single provider system in 1984-1985 provided an opportunity 
for comparison with both the Air Force 1983-84 multiple provider 
system results and the Army experience with the multiple provider 
system. The GAO found that, although the Air Force desired to 
decrease course cancellations, it experienced an overall increase 
in course cancellations, from 21 to 23 percent; while the Army 
rate rose from 28 to 30 percent. The GAO also found, hovever, 
that the Air Force cancellation rate for lower level uritr- 
graduate courses declined from 18.7 percent to 17.4 percent- 
while the Army cancellation rate rose from 28 to 30 percent! The 
GAO observed that the 17.5 percent increase in cancellations of 
vocational courses at Air Force bases was due in part to start-up 
difficulties experienced by the new provider. (The GAO also 
noted that both Services experienced increases in the rate of 
cancellation of vocational courses.) The GAO also found that 
data for the first four terms of the Air Force academic year 
1985-86 showea cancellation rates decreasing in all categories. 

The GAO reported that Air Force enrollment** ^rew by 23 percent 
from academic years 1983-84 to 1984-85, while Army enrollments 
increased 10 percent. On the other hand, the GAO found tha*: 
increases in numbers of courses increased 14 percent and 15 
percent, respectively, for the Air Force and .\rmy in these years. 
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The GAO observed that Air Force enrollment and course increases 
were partly due to the reduced required class average negotiated 
into the Air Force contract with the University of Maryland. In 
addition, the GAO found that (although the Air Force and Army do 
not use the same methods to determine such costs) Air Force 
postsecondary education support costs declined from $172,321 to 
$137,241; while Army costs declined from $3,077,673 to 
$2,896,449. The GAO found that the Services experienced 
comparable levels of performance in the delivery of 
postsecondary education m Europe as measured by such factors as 
course cancellation rates, enrollments and courses conducted. 
The GAO concluded that education services in Europe can be 
feasibly delivered using a single or multiple provider system, as 
the GAO co.TJparisop does not show that one sys*-em is more 
effective than cho other, looting Service members can earn the 
same type of degrees under either system, the GAO further 
concluded that a decision to use a single or multiple provider 
system revolves largely around the questions of whether 
competition is needed m the selection of schools and whether 
there are unmeasurable advantages to having more than one 
institution offer the same or similar lower level courses. In 
summary, the GAO concluded these are questions that should be 
resolved by the policymakers after additional experience is 
gained under the Air Force single provi'^er system. (pp. 6-7, pp. 
34-42/GAO Draft Report) 



POD RESPONSE ; Concur. Tue Depart;' at concern about the issues 
dealing with competition in the s'- ection of schools and the 
duplication of courses by both the vo-year and four -year 
institutions, will be reviewed by the DoD as it monitors the 
program. 
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August 17, 1987 



Mr. Joseph J. Eglin, Jr. 

Group Director 

Human Resources Division 

United States General Accounting Office 

Washington, DC 20548 



Re: DoD Voluntary Education: Determining and Meeting 
Postsecondarv Educational Needs in Europe — 
Comments of the City Colleges of Chi ' ago to 
the Drafc of a Proposed Report 



Dear Mr. Eglin: 



Now on p 25. 



The City Colleges of Chicago are pleased to have 
the opportunity to comment upon the draft of a proposed report 
entitled: ''DoD Voluntary Education: Determining and Meeting 
Postsecondary Educational Needs in Europe" (x^eport) . 

While the City CoXj'^qes of Chicago concur with the 
conclusion that there is no material difference between the 
Air Force and Army modes cf procurement of postsecondary 
educational services insofar as relates to the level or quality 
of services provided the users, we believe that the Report's 
findings that the Air Force sole provider mode of procurement 
has resulted in increased costs to the service and to ser- 
vicemembers and a reduction in the role of the two year schools, 
without any concomitant significant improvement in the quality 
of service delivered, compel the conclusion that the Army's 
multiple provider mode of procurement is clearly preferable 
(Report, p. 29). 

Based upon our review, we suggest that consideration 
be given tc amending the draft Report in the following areas: 

OMC AaviNiA O^ivC SUi'C >J00 k1\.kHik GCO"CiA 30341 TClC^honc (40«) 3»S HOO 
TClCCO^iC* <«04i 3«S ( 74 CA»Lt 00w«n rCLCx 
«7S^4«H4WCN0CS0U'HNC*»0flR NC.t»0«>fiOOt« r(L(»M0N((2«2>««S2 S5 

*' C4tMl3*Av S'«CC? ANNA^OklS M A • *l AN 0 7 » «0 1 Tf i f > h ON C ij 0 « ) i«3 0 0 «3 
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1. The Air Force Mode of Procurement Does Not 
Deal with *^Specific Program Areas. ^ 



The GAO report properly notes that the Air Force 
''single provider system'' was designed ''to eliminate competition 
for students between schools at the installation level • . . " 
(Draft, p. 5). However, the report errs in implying that the 
differentiation among providers is based upon "specific program 
areas." Quite the contrary, the Air Force system segments 
providers by gross level of service: all courses applicable 
towards baccalaureate degrees are provided by the sole "liberal 
arts*' provider (the University of Maryland) , leaving to the 
community college provider lower level courses that are not 
applicable towards baccalaureate degrees. Thus, the report 
correctly observes that: 



Under the Air Force's new system. Air Force personnel 
cannot complete an entire associate degree program 
at a 2-year school offering courses at an Air Force 
installation. For example, if a service member 
wants to complete an associate degree in civil 
technology from City Colleges of Chicago which is 
under contract to the Air Force, he or she must 
satisfactorily complete the 63 credits required by 
the school. Under the current contract between the 
Air Force and the 2-vear school, however^ the school 
is authorized to offer. only 22 (35 percent) of the 
recmired credits. * * * I'he f remainder of the courses! 
must be taken through the University of Maryland . 



Thus, it is erroneous to imply that the Air Force 



system segments by program area, as that term is customarily 
used to refer to the entirety of a program, such as Civil 
Technology. Rather, the Air Force mode of procurement should 
be described as segmenting service by level of provider. 



2. Most of the Differences Between the Air Force 
and Army Modes of Procurement Noted Approvingly 
in the Draft Report Arise Out of Contract 
Reguirements Independent of whether There are 
Single or Mu''tiple Providers . 

The draft Report cites approvingly a number of 



"improvements" in the Air Force mode of procurement, with the 
implication that thes.s changes from present Army procurement 
policy are related to the fact that the Air Force uses the 
sole source delivery of educational services. We suggest 



Now on pp. 26-27 



Draft, p- 32. Emphasis added. 
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that in fact the changes in procurement policy considered 
beneficial by the GAO are in most cases equally applicable 
whether the procurement is sole source or multiple provider 
in form. 

A. Multi-year contracts . Noting that the Air 
Force procurement * required a commitment on tne part of academic 
institutions to offer educational services for five years, 
the draft Report concludes that ''multi-year contracts benefit 
the servicemember and the [providing] institution" (Report, 
p. 28). The Report bases this conclusion on an analysis of 
prior practice, which th'* GAO construed as allowing institutions 
to discontinue the provision of educational services at their 
discretion. While in fact there are restrictions in the Army 
mode of procurement with regard to termination of services, 
certainly it is correct that the duration of commitment under 
that p ocurement is significantly less than what the Air 
Force now requires. 

However, regardless of whether the Procurement is 
for a sole provider or multiple providers, the Armed Services 
may reasonably require a multi-vear commitment . The validity 
of this claim will be found in the fact that the primary 
providers of services under the old Air Force mode of procure- 
ment, which, like the Army, did not entail a multi-year commit- 
ment, agreed to the five-year requirement of the new Air 
Force contract. Therefore, we suggest that consideration of 
the beneficial effects of the Air Force's multi-year contractual 
requirement shou?d be separated from consideration of the 
attributes of the sole source mode of procurement. 

We note in passing that if one looks at the years 
since the report of the Council on Postsecondary Accreditation, 
the exact same institutions which offered undergraduate programs 
in 1980 have continued to offer programs to the present day. 
Long term stability or institutional commitment does not in 
fact appear to be a matter of serious concern. But even if 
it were, the Armed Services may by the terms of the procurement 
readily require longer-term commitments, regardless of whether 
the system provides for a single institution or multiple 
providers . 

B. Inclusion of Academic Quality Controls . The 
draft Report cites approvingly provisions of the Air Force 
procurement requiring institutions to meet "specific quality- 
related requirements that had not existed in previous agreements 
used to acquire undergraduate education services" (Draft, p. 
29). These requirements include the development and implementa- 
tion of annual self-evaluation studies, assurance that instruc- 
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Now on pp. 29-35. 



tors possess academic backgrounds appropriate to instruction 
at the home campuses, and provision of library materials to 
support the courses offered. These are salutary requirements, 
and we commend the Air Force for their inclusion in the procure- 
ment. 

However, regardless of whether the p rocurement is 
for a so?e provider or multiple p roviders, the Armed Services 
may reasonably require the incl usion of such provisions in 
any procurement of educational services . Indeed, prior to 
the initiation of the Air Force mode of procurement, the 
Array's Basic Ordering Agreement (BOA) was amended to incorporate 
two of the Air Force requirements, namely the annual self- 
evaluation studies and the academic requirements for overseas 
faculty, thus accomplishing the same end within the framework 
of the multiple provider system. 

3 . The Reduction bv the Air Forc e 3,n Required 

Average Class size and Concomitant Req uirement 
to Conduct Classes When Average is M et Substan- 
tially skews the Comparative Data . 

The draft Report cites without comment the action 
taken by the Air Force in negotiating its agreements to reduce 
the required average class size from 20 to 15, and at the 
same time to require a contracting institution to conduct all 
classes once the average is met. Conversely, if the average 
is not met, thxi contract allows an institution to request 
that the installation Education Services Officer "cancel a 
sufficient number of courses to raise tht average to the 
contract minimum" (Report, p. 39) . 

First, the contractual class enrollment minimums 
are different under the Army and Air Force modes of procurement, 
thus diminishing comparability. (Depending upon the contract 
ypar, the Army minimum is either 15 or 12.) Second, and more 
significant, the manner through which the contractual minimums 
are applied under the two modes of procurement is fundamentally 
different. While under the Air Force contract the institutional 
provider is required to offer all courses at an installation 
once an average enrollment of 15 student per course is reached, 
regardless of the enrollment in any single course, under the 
Army contract an institution is required to conduct a course 
once the minimum enrollment is met for that course . While 
one may debate which contractual provision is preferable with 
regard to ensuring the maximum delivery of courses consistent 
with reasonable efficiency of operation, we suggest that the 
basic contractual difference between the ^ir Force and Army 
modes ot pro:urement invalidates any comparisons with regard 
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Mr. Joseph J. Eglin, Jr. 
August 17, 19&7 
Page 5 



to course cancellations. Notwithstanding the statistical 
implications of the d ^ f f ere nee in enrollment requirements, we 
suggest that such a contraccual provision is entirely indepen- 
dent of whether the mode of procurement calls for a single 
provider or allows for multiple providers . 

4 . The Purported Reduction in Support Costs Und er 
the Air Force Procurement Arose out of Cir~ 
cumstances Unrelated to the Mode of Procurement . 

The draft Report states that the Air Force has 
experienced a significant reduction in support costs since 
the inception of the single provider mode of procurement of 
postsecondary educational services. The Report does not, 
however, determine that the reduction is attributable to the 
sole provider system. We suggest th^,*: in fact there is no 
correlation. Rather, it is our understanding that the Air 
Force has for some time been implementing a methodical reduction 
in support services to institutional providers, including 
transportation, shipping and postage. Indeed, it appears 
that a measurable portion of the asserted reduction in Air 
Force support costs arose not as a result of a change in mode 
of procurement but rather due to funding shortages at the end 
of the fiscal year. For example, reimbursement for administra- 
tive travel has been totally eliminated during the last quarter 
of the past two fiscal years. Such budgetary constraints 
would obviously have the appearance of reducing support costs, 
without any relationship whatsoever to whether there are one 
or more providers. 

5. There is No Statistically Significant Difference 
in Course Cancellation Rates Arising out of 
the Air Force and the Army Modes of Procurement . 

The draft Report cites differences in cancellation 
rates under the two different procurement policies (Report, 
p. 36). However, the Report fails to adequately take cognizance 
of the difference in cancellation rates between the last year 
of the Air Force multiple provider mode of procurement (1983- 

84) and the first year under the sole provider mode (1984- 

85) . In the case of lower and upper division undergraduate 
courses, where the greatest complaints have been heard regarding 
course cancellations, the net change was 0.2% . This is statis- 
tically comparable to the year-to-year change in course cancel- 
lations under the multiple provider mode of procurement as 
practiced by the Army during the same period. 
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Now on pp 27-28 



6 . The Reported Increase in Course Offerings 
Under the Air F orce Mode of Procurement is 
Skewed bv the / -. rtificial Division of Respon- 
sibility Between the Two Classes of Providers . 

The draft Report notes a substantial increase in 
undergraduate courses under the Air Force mode of procurement 
and a concomitant decrease in vocational education courses 
(Report, p. 38). We suc,^est that in large measure this change 
is the result of the reclassification as "undergraduate'' of a 
large number of courses previously classified as "vocational" 
or otherwise unclassified. For example, the Chicago City 
Collegers have consistently classified all courses leading to 
an associate degree in law enforcement as vocational. However, 
under the Air Force system, law enforcement courses have been 
reclassified as part of the undergraduate general education 
program which must be offered by the four-year provider. We 
note that in the aggregate, there is no statistically sig- 
nificant difference in number of courses offered pre- and 
post-change in mode of procurement as compared to the unchanged 
Army system. 

7. Reported Difficulties in the $tart-up of Certain 
Vocational Education Programs Are the Direct 
Result of the Artificial Bifurcation of Educa- 
tional Pvograms under the Air Force Mode of 
ProcureFent . 

The Report notes difficulties encountered by the 
"new" vocational provider, the City College of Chicago, in 
starting up certain of the "more traditional vocational pro- 
grams, e.g. welding, electronics and auto mechanics" (Report, 
p. 33) . The Report further observes that prior to the bifurca- 
tion of tne procurement, the City Colleges "had offered mostly 
lower level general education requirements and vocatiorial 
programs that did not require extensive equipment and labora- 
tories." We suggest that the implication regarding the capacity 
of the institution to properly service technically complex 
programs is misplaced. Indeed, the City Colleges of Chicago 
have had long and extensive experience in the European theater 
in the delivery of highly specialized vocational programs 
requiring substantial equipment, such as those in the Emergency 
Medical Services series. But we further suggest that the 
artificial division imposed by the Air Force mode of procure- 
ment, shifting responsibility for all general education away 
from community colleges and replacing that responsibility 
with what the Report gratuitously terms "traditional" vocational 
programs such as welding and auto repair, created a dislocation 
within the overseas structure the City Colleges of Chicago 
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that could not be instantly rectified. This problem was 
exacerbated by extremely poor forecasting on the part of Air 
Force ESO's, who substantially overestimated demand for voca- 
tional courses, in some cases by a factor of three. Indeed, 
it appears that Air Force ESO's routinely characterized courses 
as "vocati*- jal" solely based on the fact that they had been 
offered by community colleges. Of course, with the sole 
provider system, a substential proportion of these courses 
were in fact reassigned to the ''liberal arts'* provider, the 
University of Maryland. 

We suggest that the relegation of community colleges 
to * traditional vocational programs" grossly distorts the 
role that these institutions have assumed over the past forty 
years. In truth, it is the nation's comm inity colleges that 
are the most substantial provider of what the Air Force defines 
as "general education," and it is the community colleges that 
have by far the roost extensive experience in providing such 
educational programs to adult learners. Leaving these institu- 
tions with "welding and auto repair" ignores the vast store 
of expertise and experience that they bring to their communities 
— and have traditionally made available to the Armed Services. 

8. The Information Adduced by the GAO Should Lead 
to the Conclusion that the Sole Provider Mode 
of Procurement is Inherently Less Cost-Effective 
than a Multiple grcvlder Mode . 

The draft Report declines to make any recommendations 
regarding the use of a sole or multiple provider mode of 
procu**ement of postsccondary educational services, and suggests 
that more time is needed for policy makers to study the Air 
Force approach (Report, pp. 7a, 42). We suggest that the 
information adduced through the study is more than sufficient 
to demonstrate the fallacy in the arguments that the multiple 
provider mode of procurement is more costly to the government 
and to the servicemembers , thac it results in excessive course 
cancellations, or that it adversely affects quality of instruc- 
tion. In fact, the Air Force mode of instruction has resulted 
in significantly higher costs to the government and to ser- 
vicemembers, it has not significantly affected course cancella- 
tions, and changes implemented by the Air Force to address 
quality issues are entirely independent of whether there are 
single or multiple providers. 
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Conclusion 

We do not disagree with the conclusion of the GAG 
that the primary factors to be addressed are "whether competi- 
tion is needed in the selection of schools and whether there 
are unmeasurable advantages to having more than one institution 
offer the same or similar lower level courses* (Report, p. 
7a). Certainly, we agree that institutions that wish to 
participate in the educational services programs of the Armed 
Services should be required to meet stringent quality standards, 
and we applaud both the Air Vorce and the Army for strengthening 
the quality control components of their respective procurements. 
But unlike moiit countries, the American system of postsecondary 
education is founded on the premise of market competition, 
with institutions actively competing to attract and hold 
students. This concept has resulted in a postsecondary system 
of unsurpassed quality and vitality, with a minimum of govern- 
ment intrusion. We contend that the GAG study in fact supports 
our position that quality and competition are not antithetical; 
rather, in the long run tney are inseparable. When one adds 
the element of price competition, the conclusion is inescapable 
that the military adaptation of the traditional American 
system of market competition, as carried out by the Army, is 
preferable to the artificial monopolies created under the Air 
Force mode of procurement. 

Gn behalf of Chancellor Rotella and President Warden, 
may we express our appreciation for the opportunity to comment 
on the draft Report, and for the extension of time granted 
the City Colleges of Chicago to obtain nfor ^tion for these 
comments from its European personnel. We coiTunena the GAG for 
the fairness and reasonableness of its observations, and the 
evenhanded nature of its recommendations. 



sincerely, / 

- 

M^iohael B. Goldstein 

Attorne 2or the 

City Colleges of Chicago 



End . 



Salvatore Rotella, Chancellor, Chicago City Colleges 
Mark Warden, President, Chicago Citywide College 



,EMC 
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Appendix IV 

GAO's Evaluation Standards Needs 
Assessment Checklist 



To evaluate each service's needs assessment and educational planning 
process, we had to determine whether the needs assessments conducted 
by the installations were consistent with established principles of needs 
assessments. In consultation with Dr. Philip L. Doughty, chairman of the 
Instructional Design, Development, and Evaluation Progran^ at Syracuse 
University, we developed an instrument to use in evaluating the scope 
and quality of educational needs assessments conducted by the military 
services in Europe. We based this **Needs Assessment Checklist'' (see 
table IV. 1) on evaluation standards developed by the Joint Committee 
on Standards for Educational Evaluation.* These are generally agreed- 
upon standards from the professional practice of evaluation. Stuf- 
flebeam suggest that the standards be applied when assessing the ade- 
quacy of needs assessments.^ 



Development of 
Checklist 



The standards developed by the Joint Committee are grouped under 
four attributes or categories of evaluation: utility, feasibility, propriety, 
and accuracy. For each attribute, we selected, with the assistance of the 
consultant, the standards that best applied to the educational needs 
assessment context. We then developed questions to measure specific 
indexes of each standard. A positive response to a question indicated 
that the standard was being addressed in a service's needs assessment 
process. Table IV. 1 presents each attribute, its definition, the standards 
that apply to the needs assessment process, and examples of the ques- 
tions we developed for that standard. 



*The Joint Committee on Standards for Educational Evaluation, Standards for Evaluations of Educa- 
tional Programs, Projects^ and Materials (New York: McGraw-Hill Book Co., 1981). 

^Daniel L Stuf flebeam, et al.. Conducting Educational Needs Assessments (Kluwer-Nghoff Publish- 
ing, 1985), p. 181. 
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Table 1V.1: Needs Assessment Checklist 



Needs 
assessment 
attribute and 
definition 



Standards and sample questions 



Utility: To what 
extent does the 
needs assessment 
process focus on 
the appropriate 
population, address 
questions, obtain 
relevant 

information, report 
it clearly, and assist 
in the use of the 
information? 



Audience identification • Audiences involved in or affected by the 
needs assessment should be identified so that their information needs 
can be addressed. 

1. Does the needs assessment process identify the audiences that are 
involved in or affected by the voluntary postsecondary education 
program? 

2. Does the needs assessment report state why these audiences are 
important? 

Needs assessor credibility - Those conducting needs assessment 
should be competent to perform the assessment so that their findings 
achieve maximum credibility and acceptance. 

1 . Can you determine who conducted the needs assessment? 

2. Is there evidence that the person who conducted the needs 
assessment has experience in this area (e.g., educational background, 
job experience, needs assessment experience)? 

3. Are resource materials (guides, handbooks, examples of surveys, 
etc.) available that can be used to help develop the needs 
assessment? 

^. If yes, is there evidence that these materials were used in the needs 
assessment process? 

5. Are there any military or nonmilitary staff who provide assistance or 
advice in designing and conducting the needs assessment? 

Information scope and selection - Information collected should be of 
such scope and selected in such ways as to address pertirient 
questions about the object of the needs assessment. 

1 . Are data collected from the following groups concerning their 
preferences for either specific programs or courses? 

Ty pe of group 

a. Military servicemembers 

b. Civilian DOD employees 

c. Adult family members 

d. Base command 

e. Others (specify) 

Enrollment status 

f. Currently enrolled students 

g. Recently enrolled students 

h. All eligible persons 

(continued) 
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Needs 
assessment 
attribute and 
definition 



Standards and sample questions 



Utility 

(continued) 



2. If yes, do the preference questions/data focus on: 



a. Course preference only 

b. Program preference only 

c. Both course and program preference 

d. Course scheduling limes 

e. Type of degree (i.e., associate, bachelors, certificate) 

f. Service-related skills, knowledge, and abilities (SKAs) 

g. Personally desired SKAs 

h. Other (specify) 

3. Are data collected on how the educational program can help 
improve the lives of eligible participants? 

4. Are data collected from the military command on how the 
educational program can help improve job performance and 
promotability? 

5. Are longitudinal or histoncal data collected on: 

a. Programs 

b. Individuals 

6. Are data on base population characteristics such as educational 
level/background, military/nonmilitary status, etc.. collected? 

7. Are the data collected in each category relevant to the overall 
objectives of the needs assessment? 

a. Preference data 

b. Longitudinal or histoncal data 

c. Population characteristics, data 

d. SKA d-'ta 

Report clarity • The needs assessment report should describe the 
group whose needs are being assessed and its location, and the 
purposes, procedures, and findings to ensu-e understanding of what 
was done, rationale, data sources, conclusions, and recommendations. 

1. Is there a final report or summary of results on the needs 
assessmcnf? 

2. Are technical terms defined? 

3. A;e data displays understandable? 

4. Does the report describe the characteristics of the population 
whose neeas are being assessed? 

5. Are the purposes of the needs assessment described in the report 
(i.e.. why the needs assessment was conducted)? 

6. Are the procedures followed during the needs assessment process 
descnbed in the report (i.e.. what was done)? 



(continued) 
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Needs 
assessment 
attribute and 

definition Standards and sample questions 

Utility Report dissemination - Needs assessment findings should be 

(continued) disseminated to clients and other right-to-know audiences. 



1 . Are the findings of the needs assessment made available to: 

a. Military servicemembers 

b. Civilian DOD employees 

c. Adult family members 

d. Base command 

e. Headquarters command 

f. School representatives 

g. Others (specify) 

Report timeliness - Release of reports should be timely, so that 
audiences can best use the findings. 

1 . Are needs assessments conducted on an annual basis? 

2. If not annually, are they conc'ucted on a regular basis? 

3. Is the needs assessment available before educational planning is 
completed by the base? 

4. Is the needs assessment available before planning is completed jy 
the schools? 

5. Is the information from the needs assessment used in program 
development? 

Impact ■ Needs assessments should be planned and conducted in 
ways to encourage folio yv-through by members of the audience. 

1 . Is the needs assessment required by the branch of the military? 

2. Is the needs assessment required by the base command? 

3. Is the needs assessment conducted in accordance with the 
requirements of the military services? 

4. At the service level, are there requirements stating that the needs 
assessment be used in educational planning? 

5. At the command level, are there requirements stating that the needs 
assessment be used in educational planning? 

6. At the base level, are there requirements stating that the needs 
assessment be used in educatio.ial planning? 

7. Is there evidence that the information obtained in the needs 
assessment process is used ir» educational planning? 

8. Is there evidence that the needs assessment influences program 
and course offerings'? 



(continued) 
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Needs 
assessment 
attribute and 

definition Standards and sample questions 

Feasibility: To Practical procedures * The needs assessment procedures should bi 
what extent is the practical, so disruption is kept to a minimum and required Information 
needs assessment can be obtained, 
process cost- 
effective and 1 . Are the ESOs experiencing any of the following problems in 
workable in real administering the needs assessment? 
world settings? Is 

the process a. Insufficient time 

operable in the b. Insufficient other resources 

setting in which is it c. Analysis problems 

to be applied and is d. Reporting problems 

it realistic, prudent, e. Access to respondents' problems 

diplomatic, f. Access to data problems 

politically viable, g. Unclear definitions 



1. Respont^ rate and/or representativeness problems 
j. Other (specify) 

Political viability • Tne needs assessment process should have 
sufficient support so that its purposes can be achieved. 

1 Is there a statement, either written or stated in an interviev.. of the 
objectives of the base's voluntary postsecondary undergraduate 
education program? 

2. Does there appear to be a consensus as to the objectives of the 
base's program by the .arlous groups involved in the program'^ 

3. Is there evidence that the various grouDS involved in the base's 
program agree as to the purpose of the needs assessment? 

4. Are there requirements that staff other than the ESO participate in 
the plannino/implementing of the needs assessment? 

5. In the ESQ'S opinion, does the military command s» jcture regard 
needs assessment as an important task? 

Cost effectiveness * The needs assessment should provide 
information of sufficient value to justify the resources expended. 

1 Does the ESO have adequate, in his/her opinion, resources to plar. 
and conduct the needs assessment? 

2. Does t** " base command believe that the results of the needs 
assessmo . are worth the time and otner resources required to 
conduct the study? 

3. Does the ESO believe that the results of the needs assessment are 
worth the time arid other resources required to conduct the study? 



and frugal? 



h. Unclear rraterials 



(continued) 
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Needs 
assessment 
attribute and 
definition 



Standards and sample questions 



Accuracy: To what 
extent has the 
needs assessment 
produced sound 
information? Are 
the data technically 
adequate and are 
the conclusions 
derived logically 
from the data? 



Group identification > The group whose needs are being assessed 
should be sufficiently examined so that it can be clearly identified and 
characterized. 

1. Are the beneficiaries of the educational program represented in the 
needs assessment? 

Type of group 

a. Military servicemembers 

b. Civilian DOD employees 

c. Adult family members 

d. Base command 

e. Others (specify) 

Enrollment status 

i. Currently enrolled students 

g. Recently enrolled students 

h. All eligible persons 

Context analysis • The context in which the popuiatiori exists should 
be examined in enough detail so that its liKely influences on the 
population can be identified. 

1. Are data detailing basic demographic features of the military 
personnel on the base presented in the needs assessment? 

2. Does the base provide an incentive, other than tuition support, to 
the military personnel to participate in the program? 

3. Does the base command direct, in any way. the selection of 
programs or courses that individual military personnel take'^ 

4. Is the mission of the base described in the needs assessment? 

Described purposes and procedures • The purposes and 
procedureb of the needs assessment should be monitored and 
described in enough detail that they can be identified and assessed. 

1. Does any group or individual with authonty over the ESQ monitor the 
planning, implementation, and reporting of the needs assessment? 

2. Is there any wptten guidance available to the ESOs on methods of 
conducting a needs assessment? 

3. Is there a descnption addressing how the needs assessment data 
will be ust'^d in the educational planning process? 

(continued) 
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Needs 
assessment 
attribute and 

definit ion Standards and sarrpie question s 

Accuracy 4. Is there a description addressing when the needs assessment data 

(continued) will be used in the educational planning process? 

Pefensible information sources - The sources of all information 
should be described in enough detail that the adequacy of the data 
can be assessed. 

1 . Does the needs assessment report indicate the size of the 
population eligible to participate? 

2. Does the needs assessment report indicate the number of 
individuals from whom data were collected? 

3. Is there a description of the specific data collection methods? 

4. Are sampling procedures used? 

5 If the report does not indicate whether their samples are 
representative or not, is there evidence that the samples are 
representative from other sources? 

Validity - The information-gathering instruments anu procedures 
should be chosen or developed and then implemented in ways that 
will insure that the interpretation arrived at is valid for the given use. 

1 Is there a correspondence between the types of data being 
collected and the overall oojective of the needs assessment? 

2. Is there more than one measurement method (e.g., survey and 
interview data review) used to assess educational needs? 

3. Is there a statement as to what specific data are being used in the 
assessment process? (e.g., "what courses servicemembers want to 
take" or "program/area preference of study") 

4. Are the data being used ;n the assessment process of the same 
type as the data being reported? (e.g., measuring course preference 
and reporting program preference, or measuring cancellations and 
reporting course dropouts) 

Reliability - The information-gathering instruments and procedures 
should be chosen or developed and then implemented in ways that 
will insure the information obtained Is sufficiently reliable for the 
intended use. 

1. Is there evidence that the instruments used in the needs 
assessment process were developed through a systematic instrument 
development process? 

2. If more than one method of data collection was used to assess the 
same concept (e.g., assessing program preference with both survey 
and interviews), are the results similar? 

3. Aia the instruments and procedures ubcd In the needs assessment 
process standardized (i.e., used each time an assessment is 
conducted on that base)? 

(continued) 
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Needs 
assessment 
attribute and 

definition Standards and sample questions 

Accuracy Systematic data control - The data collected, ore :essed, and 

(continued) reported in a needs assessment should be reviewed and corrected, If 

necessary, so that the data/report of the needs assessment will not be 

flawed. 

1 . Is there evidence that the data heing collected for the needs 
assessment are 

a. reviewed by other officials, 

b. 'hecked for errors in recording resultL< and 

c. collected according to prescribed procedures? 

Justified conclusions > The conclusions reached In a needs 
assessment should be explicitly justified, so that the audiences can 
assess them. 

1 . Do the conclusions of the needs assessment match the objectives 
of the needs assessment? 

2. Are the conclusions supported by the evidence presented in the 
needs assessment? 

3. Do the recommendations fcHcw quickly from the conclusions? 

4. Does the needs assessment report indicate any limitations of the 
needs assessment (I.e., what was excluded, cautions, etc.)? 

Objective reporting - The needs assessment procedures should 
provide safeguards to protect the needs assessment findings and 
reports against distortion by the personal feelings and biases of any 
party to the needs assessment. 

1 . Are the needs assessment findings and report reviewed by other 
officials (other than th * individual who conducted the assessment)? 



Nine experts (see app. V) in the needs assessment and evaluation fields 
reviewed the Needs Assessment Checklist. They generally agreed that 
the evaluations standards developed by the Joint Committee were appli- 
cable to our purposes and that our checklist questions addressed the 
standards. Where appropriate, we revised the checklist in response to 
the reviewers' comments. 

With the assistance of the consultant, we conducted a training session in 
which we reviewed the items for each standard with the GAG staff 
assigned to complete the checklist. Topics covered includea (1) clarifica- 
tion of terms used in the items, (2> type of data that could be used to 
support a response to an item, and (3) procedures used to document the 
support for each response. 
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Use of Checklist ^^^^ "^^^ complete the checklist were collected at the 10 locations in 

Europe. The data included (1) responses to interview questions from 
relevant parties, (2) documents collected from both the institutions pro- 
viding educational services and service officials in Europe and the 
United States, and (3) information gained from records at each of the 10 
locations. 

Two GAG staff members independently completed a checklist for 3 of the 
10 locations. A third GAG staff person reviewed both checklists for the 
three locations to determine the degree rf consistency between the two 
raters. Once the judgement was made that a checklist could be com- 
pleted with consistency across differe/it raters, one of the two raters 
then completed the remaining seven locations. 
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— Dr. Michael S. Caldwell , Bureau of Educational Research, School of 
Education, University of Virginia 

— Dr. Gary Gottfredson , Center for Social Organization of Schools, John 
Hopkins University 

— Dr. Conrad Katzenmeyer , Office of Educational Research and 
Improvement, U.S. Department of Education 

— Dr. Roger Kaufman , Center for Needs Assessment and Planning, Flor- 
ida State University 

— Dr. Charles McClintock, College of Human Ecology, Cornell 
University 

— Dr. Jason Millman , Department of Education, College of Agriculture 
and Life Sciences, Cornell University 

— Dr. Nick Smith , Instructional Design, Development and Evaluation, 
Syracuse University 

— Dr. Daniel L. Stufflebeam , Evaluation Center, Western Michigan 
University 

— Dr. William Trochim , Department of Human Service Studies, College 
of Human Ecology, Cornell University 
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Degree to Which Services Met Standards for 
Needs Assessment Adequacy 



rhe degree to which standards for needs assessment adequacy were met 
by the individual services is shown in table VI.l. The percentages repre- 
sent the number of positive responses to requirements included in the 
questions we developed for each standard. For example, the standard 
^^information scope and selection" has 25 requirements with a total of 
25 possible yes/no responses. The Army, with 4 locations reviewed, 
would have a possible 100 responses— 4 locations times 25 require- 
ments. In completing the needs assessment che:::Iist for the Army loca- 
tions, the Army had 79 out of the 100 possible responses rated yes, thus 
the Army scored 79 percent on this standard. The Air Force, on the 
other hand, scored 74 percent for this standard because it received 92 
positive responses to a possible 125 responses— 5 locations times 25 
requirements. 



Table V1.1: Adequacy of Needs 

Assessment by Three Services (Survey 
Responses) 



Needs 



Positive responses (percent) 



assessment 
attribute 


Standard 


Air 
Force 


Army 


Navy 


Collectively 


Utility 


Audience identification 


75 


88 


78 


82 




Needs assessor credibility 


88 


90 


80 


88 




Information scope and selection 


74 


79 


88 


77 




Report clarity 


78 


72 


50 


73 




Report dissemination 


42 


39 


55 


42 




Report timeliness 


76 


75 


40 


70 




Impact 


74 


82 


88 


82 


Feasibility 


Practical procedures 


28 


58 


6C 


33 




Political viability 


74 


66 


86 


70 




Cost effectiveness 


20 


9 


0 


13 


Accuracy 


Group Identification 


80 


77 


86 


79 




Context 'analysis 


55 


44 


50 


40 




Decoribe J purposes and 
procedures 


69 


89 


56 


76 




Derensible information sources 


69 


77 


71 


73 




Validity 


78 


100 


88 


88 




R(;iiability 


20 


83 


33 


47 




Systematic data control 


73 


83 


33 


70 




Justified conclusions 


55 


81 


30 


65 




Objective reporting 


95 


50 


75 


60 
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